Nov 03 07:52:56.452723 ip-10-0-140-232 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 07:52:56.827762 ip-10-0-140-232 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:52:56.827762 ip-10-0-140-232 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 07:52:56.827762 ip-10-0-140-232 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:52:56.827762 ip-10-0-140-232 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:52:56.832147 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832067    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.832147 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832147    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.832147 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832151    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832156    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832159    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832162    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832164    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832167    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832170    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832173    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832175    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832178    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832181    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832184    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832186    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832189    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832192    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832194    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832197    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832200    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832202    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832210    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.832240 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832213    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832216    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832218    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832221    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832224    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832227    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832229    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832232    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832235    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832237    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832240    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832243    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832246    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832248    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832251    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832256    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832260    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832263    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832267    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.832709 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832269    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832272    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832275    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832278    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832281    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832283    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832286    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832288    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832291    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832293    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832296    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832299    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832301    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832304    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832306    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832309    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832312    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832314    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832317    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.833286 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832319    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832322    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832324    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832327    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832330    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832332    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832335    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832338    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832340    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832343    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832345    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832348    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832351    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832354    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832356    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832359    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832362    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832365    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832367    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832370    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.833743 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832372    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832377    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832380    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832383    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832385    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832388    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832391    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832393    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832396    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832399    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832401    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832404    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832420    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832423    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832426    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832429    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832432    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832434    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832437    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.834219 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832440    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832442    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832445    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832448    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832450    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832453    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.832456    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.833991    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.833999    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834002    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834005    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834008    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834011    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834014    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834016    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834019    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834021    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834024    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834027    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834029    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.834661 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834032    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834035    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834038    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834041    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834043    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834045    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834048    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834051    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834054    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834057    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834060    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834062    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834065    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834068    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834071    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834073    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834076    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834078    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834081    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834085    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.835128 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834088    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834090    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834093    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834096    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834098    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834101    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834104    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834106    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834109    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834112    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834115    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834117    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834120    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834124    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834127    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834130    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834132    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834136    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834140    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834143    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.835646 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834145    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834148    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834151    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834155    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834158    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834161    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834164    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834167    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834169    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834172    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834175    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834177    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834180    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834183    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834186    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834188    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834191    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834193    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.836109 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834196    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834198    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834201    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834205    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834207    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834210    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834213    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834215    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834219    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834221    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834224    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834227    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834229    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834232    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834235    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834238    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834240    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834243    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834245    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834248    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.836542 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834251    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834254    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834257    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834259    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834261    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834264    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834266    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834269    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834272    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834274    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834277    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834279    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834282    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834284    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834287    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834841    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834846    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834849    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834852    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834855    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.837008 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834858    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834861    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834863    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834866    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834869    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834871    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834874    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834877    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834879    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834882    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834884    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834887    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834890    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834892    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834895    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834897    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834900    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834903    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834905    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.837502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834909    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834911    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834914    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834917    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834919    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834922    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834924    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834927    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834930    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834933    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834935    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834938    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834941    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834943    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834946    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834948    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834952    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834956    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834959    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.837953 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834961    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834964    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834966    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834969    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834971    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834974    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834976    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834979    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834982    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834984    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834987    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834990    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834993    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834996    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.834999    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835002    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835004    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835007    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835009    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835012    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.838403 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835014    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835017    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835019    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835022    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835024    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835027    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835029    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835032    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835034    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835036    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835039    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835041    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835044    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835046    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835049    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835051    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835054    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835056    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835059    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835061    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.838879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835063    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835066    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835069    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835072    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835074    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835077    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835080    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835088    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835091    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835093    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835096    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835098    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835101    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835104    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835106    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835109    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835111    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835114    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835116    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.839347 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835121    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835133    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835137    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835140    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835245    2574 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835252    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835259    2574 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835264    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835268    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835272    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835276    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835283    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835287    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835290    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835294    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835297    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835300    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835303    2574 flags.go:64] FLAG: --cgroup-root=""
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835306    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835309    2574 flags.go:64] FLAG: --client-ca-file=""
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835312    2574 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835316    2574 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835320    2574 flags.go:64] FLAG: --cluster-domain=""
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835323    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 07:52:56.839796 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835326    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835329    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835332    2574 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835337    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835340    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835343    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835346    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835349    2574 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835352    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835355    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835358    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835361    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835365    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835368    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835371    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835374    2574 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835377    2574 flags.go:64] FLAG: --enable-server="true"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835380    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835385    2574 flags.go:64] FLAG: --event-burst="100"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835389    2574 flags.go:64] FLAG: --event-qps="50"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835392    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835397    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835400    2574 flags.go:64] FLAG: --eviction-hard=""
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835419    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 07:52:56.840349 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835422    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835425    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835428    2574 flags.go:64] FLAG: --eviction-soft=""
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835431    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835434    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835437    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835441    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835444    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835447    2574 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835450    2574 flags.go:64] FLAG: --feature-gates=""
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835454    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835457    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835460    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835464    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835467    2574 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835470    2574 flags.go:64] FLAG: --help="false"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835473    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835476    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835479    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835482    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835485    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835489    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835492    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 07:52:56.840966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835494    2574 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835498    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835501    2574 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835504    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835508    2574 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835511    2574 flags.go:64] FLAG: --kube-reserved=""
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835515    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835518    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835521    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835524    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835527    2574 flags.go:64] FLAG: --lock-file=""
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835530    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835533    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835536    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835542    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835545    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835548    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835552    2574 flags.go:64] FLAG: --logging-format="text"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835555    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835558    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835561    2574 flags.go:64] FLAG: --manifest-url=""
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835564    2574 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835569    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835572    2574 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835576    2574 flags.go:64] FLAG: --max-pods="110"
Nov 03 07:52:56.841573 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835578    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835582    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835584    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835587    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835591    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835594    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835597    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835603    2574 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835606    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835609    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835612    2574 flags.go:64] FLAG: --pod-cidr=""
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835617    2574 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835620    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835624    2574 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835627    2574 flags.go:64] FLAG: --port="10250"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835630    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835633    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b7e0d9248434f705"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835637    2574 flags.go:64] FLAG: --qos-reserved=""
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835640    2574 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835643    2574 flags.go:64] FLAG: --register-node="true"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835646    2574 flags.go:64] FLAG: --register-with-taints=""
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835650    2574 flags.go:64] FLAG: --registry-burst="10"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835652    2574 flags.go:64] FLAG: --registry-qps="5"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835655    2574 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835659    2574 flags.go:64] FLAG: --reserved-memory=""
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835663    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 07:52:56.842161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835666    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835669    2574 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835672    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835675    2574 flags.go:64] FLAG: --runonce="false"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835678    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835681    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835684    2574 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835687    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835690    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835693    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835696    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835699    2574 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835702    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835704    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835707    2574 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835710    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835713    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835717    2574 flags.go:64] FLAG: --system-cgroups=""
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835720    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835726    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835728    2574 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835731    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835735    2574 flags.go:64] FLAG: --tls-min-version=""
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835738    2574 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835740    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 07:52:56.842784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835743    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835746    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835749    2574 flags.go:64] FLAG: --v="2"
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835754    2574 flags.go:64] FLAG: --version="false"
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835758    2574 flags.go:64] FLAG: --vmodule=""
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835762    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.835765    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835921    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835925    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835928    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835931    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835933    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835936    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835938    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835941    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835943    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835946    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835950    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835953    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835956    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835959    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.843354 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835961    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835963    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835966    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835970    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835973    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835975    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835978    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835981    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835983    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835986    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835988    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835991    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835994    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835996    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.835999    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836002    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836004    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836007    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836010    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836012    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.843859 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836015    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836017    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836020    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836022    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836025    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836027    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836030    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836032    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836035    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836037    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836040    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836042    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836045    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836047    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836050    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836054    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836057    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836060    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836062    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.844353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836065    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836067    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836070    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836072    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836075    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836078    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836080    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836083    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836085    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836088    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836090    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836093    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836096    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836098    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836101    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836103    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836106    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836109    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836111    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836114    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.844811 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836117    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836119    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836122    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836124    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836127    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836139    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836143    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836146    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836149    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836152    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836155    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836158    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836160    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836163    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836165    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836168    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836170    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836173    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836176    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.845280 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836178    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836182    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836186    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836189    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836192    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836194    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836197    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836200    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836202    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836205    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836208    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836210    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836213    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836216    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.836241    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836276    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.845784 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836281    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836284    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836287    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836290    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836293    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836297    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836301    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836305    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836307    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836310    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836313    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836316    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836318    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836321    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836323    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836326    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836329    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836331    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836334    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.846167 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836337    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836339    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836342    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836344    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836348    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836350    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836353    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836356    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836360    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836363    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836366    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836369    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836371    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836374    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836376    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836379    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836381    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836384    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836386    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.846629 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836391    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836394    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836397    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836400    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836402    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836418    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836421    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836423    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836426    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836429    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836431    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836434    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836436    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836439    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836442    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836444    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836447    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836450    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836453    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836456    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.847084 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836458    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836461    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836463    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836466    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836468    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836471    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836473    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836476    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836478    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836482    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836485    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836487    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836491    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836495    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836498    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836500    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836503    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836506    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836508    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836511    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.847570 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836514    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836516    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836519    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836521    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836524    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836526    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836529    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836532    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836534    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836537    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836539    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836542    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836545    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836548    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836550    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836553    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836556    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836558    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836561    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836564    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.848038 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836567    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836569    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836572    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836574    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836577    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836581    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.836584    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.843057    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.843072    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.843678    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.843684    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843745    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843751    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843754    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843757    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843760    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843763    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843766    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843768    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.848513 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843771    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843774    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843776    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843779    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843782    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843785    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843787    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843790    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843792    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843795    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843798    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843800    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843803    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843805    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843808    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843810    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843813    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843815    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843819    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.848945 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843822    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843825    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843827    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843830    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843832    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843835    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843838    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843840    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843843    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843846    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843848    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843851    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843853    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843856    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843859    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843861    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843864    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843866    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843869    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843872    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.849502 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843876    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843881    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843884    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843887    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843889    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843891    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843894    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843897    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843899    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843901    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843905    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843908    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843910    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843913    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843915    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843918    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843921    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843923    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843926    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843928    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.850806 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843931    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843933    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843936    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843938    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843940    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843943    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843946    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843950    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843954    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843957    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843960    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843963    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843966    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843969    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843971    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843974    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843976    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843979    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843981    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843983    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.851377 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843986    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843988    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843991    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843994    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843996    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.843999    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844002    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844004    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844007    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844009    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844012    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844014    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844017    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844020    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844022    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844024    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844027    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844029    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.851992 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844032    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.844059    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844214    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844219    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844223    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844226    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844229    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844232    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844235    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844238    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844240    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844243    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844245    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844248    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844250    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.852737 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844253    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844255    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844258    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844260    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844263    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844266    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844268    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844270    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844273    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844275    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844278    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844280    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844283    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844285    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844287    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844290    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844292    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844295    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844297    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844300    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.853199 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844302    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844305    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844307    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844310    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844313    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844315    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844318    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844321    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844323    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844326    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844329    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844331    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844334    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844336    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844339    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844341    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844344    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844346    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844348    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844351    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.853879 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844353    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844356    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844359    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844361    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844364    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844366    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844369    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844371    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844374    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844376    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844379    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844381    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844384    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844386    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844389    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844391    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844393    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844397    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844399    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844402    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.854382 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844404    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844419    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844422    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844425    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844427    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844430    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844433    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844435    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844438    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844440    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844443    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844445    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844447    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844450    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844452    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844455    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844458    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844460    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844462    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844465    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.855095 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844467    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844470    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844473    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844475    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844478    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844480    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844482    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844485    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844488    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844491    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844496    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844500    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844503    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.844526    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844559    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:56.855587 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844563    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844566    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844570    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844573    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844576    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844578    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844581    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844584    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844586    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844589    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844591    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844594    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844596    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844599    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844601    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844604    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844607    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844609    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844613    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:56.855938 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844615    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844618    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844620    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844623    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844625    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844628    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844630    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844633    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844635    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844638    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844641    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844643    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844646    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844648    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844651    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844653    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844656    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844658    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844661    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844663    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:56.856373 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844666    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844668    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844671    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844673    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844675    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844678    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844680    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844683    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844685    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844689    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844691    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844694    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844696    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844699    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844701    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844704    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844706    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844709    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844711    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:56.856861 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844714    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844716    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844718    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844721    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844724    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844726    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844729    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844731    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844734    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844736    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844739    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844741    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844743    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844746    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844749    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844751    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844756    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844759    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844762    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844764    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:56.857355 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844767    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844770    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844772    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844775    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844778    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844780    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844783    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844785    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844788    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844790    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844793    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844796    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844798    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844801    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844803    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844806    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844809    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844811    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844814    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844817    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:56.857913 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844819    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844822    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844825    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844828    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844833    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844837    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:56.844840    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.844891    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.846785    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.847596    2574 server.go:1014] "Starting client certificate rotation"
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.847697    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.847735    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.849361    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:52:56.858586 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.852946    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:52:56.863838 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.863822    2574 log.go:25] "Validated CRI v1 runtime API"
Nov 03 07:52:56.868992 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.868974    2574 log.go:25] "Validated CRI v1 image API"
Nov 03 07:52:56.869930 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.869898    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 07:52:56.873382 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.873358    2574 fs.go:135] Filesystem UUIDs: map[1ae67e76-f702-45f9-9d69-3355f221c940:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eb314d51-fa80-4249-aa78-95ede7f6766a:/dev/nvme0n1p3]
Nov 03 07:52:56.873494 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.873381    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 07:52:56.878879 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.878755    2574 manager.go:217] Machine: {Timestamp:2026-11-03 07:52:56.877619369 +0000 UTC m=+0.318315508 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3064135 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2691129ce845463d813083a90a6b63 SystemUUID:ec269112-9ce8-4546-3d81-3083a90a6b63 BootID:d9667ff0-041e-41ab-82fd-e90a4b32b392 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:3b:ab:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:3b:ab:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:8e:7c:ca:a4:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 07:52:56.878879 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.878876    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 07:52:56.879044 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.878989    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 07:52:56.881882 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.881852    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 07:52:56.882056 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.881884    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-232.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 07:52:56.882145 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.882061    2574 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 07:52:56.882145 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.882074    2574 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 07:52:56.882145 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.882109    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:52:56.882145 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.882131    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:52:56.882874 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.882861    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 07:52:56.883275 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.883253    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 07:52:56.886430 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.886394    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 07:52:56.886561 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.886549    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 07:52:56.886620 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.886596    2574 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 07:52:56.886620 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.886611    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 07:52:56.886704 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.886650    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 07:52:56.886704 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.886664    2574 kubelet.go:405] "Adding apiserver pod source"
Nov 03 07:52:56.886704 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.886692    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 07:52:56.887392 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.887377    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:52:56.889657 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.889641    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 07:52:56.894960 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.894943    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:52:56.895753 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.895735    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 07:52:56.895820 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.895782    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 07:52:56.896016 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.895975    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 07:52:56.896016 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.895992    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 07:52:56.896016 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896001    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 07:52:56.896016 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896010    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 07:52:56.896016 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896016    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 07:52:56.896153 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896022    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 07:52:56.896652 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896643    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 07:52:56.896681 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896653    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 07:52:56.896681 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896661    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 07:52:56.896681 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896667    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 07:52:56.896681 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896676    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 07:52:56.896798 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.896687    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 07:52:56.898181 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.898164    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 07:52:56.898181 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.898181    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 07:52:56.898786 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.898769    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7zvn4"
Nov 03 07:52:56.901970 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.901955    2574 server.go:1263] "Started kubelet"
Nov 03 07:52:56.902079 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.902029    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 07:52:56.902122 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.902059    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 07:52:56.902163 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.902147    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 07:52:56.902862 ip-10-0-140-232 systemd[1]: Started Kubernetes Kubelet.
Nov 03 07:52:56.903428 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.903173    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 07:52:56.904635 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.904621    2574 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 07:52:56.905931 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.905905    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7zvn4"
Nov 03 07:52:56.906851 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.906828    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 07:52:56.908419 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.908392    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 07:52:56.908476 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.908399    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 07:52:56.908547 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.908532    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:56.908912 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.908895    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 07:52:56.908912 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.908898    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 07:52:56.908912 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.908914    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 07:52:56.909074 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.908990    2574 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 07:52:56.909074 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.908997    2574 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 07:52:56.910900 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.910883    2574 factory.go:153] Registering CRI-O factory
Nov 03 07:52:56.910900 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.910899    2574 factory.go:223] Registration of the crio container factory successfully
Nov 03 07:52:56.911027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.910939    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 07:52:56.911027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.910945    2574 factory.go:55] Registering systemd factory
Nov 03 07:52:56.911027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.910952    2574 factory.go:223] Registration of the systemd container factory successfully
Nov 03 07:52:56.911027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.910964    2574 factory.go:103] Registering Raw factory
Nov 03 07:52:56.911027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.910981    2574 manager.go:1199] Started watching for new ooms in manager
Nov 03 07:52:56.911662 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.911651    2574 manager.go:319] Starting recovery of all containers
Nov 03 07:52:56.913469 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.913439    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-232.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 07:52:56.913571 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.913496    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-232.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 07:52:56.913571 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.913550    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-232.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 07:52:56.913679 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.913591    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 07:52:56.914703 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.912859    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-232.us-east-2.compute.internal.18e479b6a1909491  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-232.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-232.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 07:52:56.901915793 +0000 UTC m=+0.342611932,LastTimestamp:2026-11-03 07:52:56.901915793 +0000 UTC m=+0.342611932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-232.us-east-2.compute.internal,}"
Nov 03 07:52:56.915072 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.915052    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:56.921724 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.921670    2574 manager.go:324] Recovery completed
Nov 03 07:52:56.926215 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.926200    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:56.928078 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.928062    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:56.928153 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.928093    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:56.928153 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.928106    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:56.928660 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.928648    2574 cpu_manager.go:226] "Starting" policy="none"
Nov 03 07:52:56.928660 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.928658    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 07:52:56.928778 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.928673    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 07:52:56.928778 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.928712    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:56.929341 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.929321    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:56.929429 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.929356    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:56.929429 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.929369    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:56.930709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.930697    2574 policy_none.go:50] "Start"
Nov 03 07:52:56.930743 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.930712    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 07:52:56.930743 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.930725    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 07:52:56.932688 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.932677    2574 policy_none.go:44] "Start"
Nov 03 07:52:56.963997 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.963966    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 07:52:56.966642 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.966491    2574 manager.go:341] "Starting Device Plugin manager"
Nov 03 07:52:56.966738 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.966667    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 07:52:56.966738 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.966681    2574 server.go:87] "Starting device plugin registration server"
Nov 03 07:52:56.966903 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.966891    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 07:52:56.966936 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.966905    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 07:52:56.967031 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.967013    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 07:52:56.967117 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.967106    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 07:52:56.967166 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:56.967118    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 07:52:56.967709 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.967685    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 07:52:56.967776 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:56.967743    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.056510 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.056480    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 07:52:57.058553 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.057815    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 07:52:57.058553 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.057832    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 07:52:57.058553 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.057858    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 07:52:57.058553 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.057892    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 07:52:57.060902 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.060883    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:57.068007 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.067993    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:57.068781 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.068767    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:57.068831 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.068794    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:57.068831 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.068808    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:57.068831 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.068831    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.075696 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.075681    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.075768 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.075708    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-232.us-east-2.compute.internal\": node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.084467 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.084382    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.158847 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.158799    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"]
Nov 03 07:52:57.158987 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.158885    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:57.159888 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.159872    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:57.159989 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.159905    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:57.159989 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.159919    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:57.162215 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.162195    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:57.162369 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.162354    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.162433 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.162383    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:57.162920 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.162905    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:57.163000 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.162937    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:57.163000 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.162951    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:57.163129 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.163032    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:57.163129 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.163056    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:57.163129 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.163066    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:57.165177 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.165162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.165253 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.165189    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:57.166218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.166201    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:57.166337 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.166232    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:57.166337 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.166249    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:57.184863 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.184847    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.192652 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.192636    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found" node="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.197480 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.197465    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found" node="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.211271 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.211250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f16205502b8d2684b7e492677380c9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"40f16205502b8d2684b7e492677380c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.211340 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.211274    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.211340 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.211291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.285708 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.285679    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.312109 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.312081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f16205502b8d2684b7e492677380c9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"40f16205502b8d2684b7e492677380c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.312233 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.312114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.312233 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.312130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.312233 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.312156    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.312233 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.312193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f16205502b8d2684b7e492677380c9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"40f16205502b8d2684b7e492677380c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.312233 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.312193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.386392 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.386314    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.486968 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.486933    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.500143 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.500115    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.501213 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.501195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:57.587348 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.587315    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.687884 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.687813    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.788353 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.788317    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.819107 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.819083    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:57.847499 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.847469    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 07:52:57.847936 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.847610    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 07:52:57.847936 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.847649    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 07:52:57.847936 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.847654    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 07:52:57.888892 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.888860    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:57.912237 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.912178    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 07:47:57 +0000 UTC" deadline="2028-06-26 15:42:02.113833813 +0000 UTC"
Nov 03 07:52:57.912237 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:57.912230    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14431h49m4.201606324s"
Nov 03 07:52:57.989287 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:57.989203    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:52:58.062463 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:58.062382    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:58.106443 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:58.106391    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15f1c78c19d0b39887f68042df09ebe.slice/crio-7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e WatchSource:0}: Error finding container 7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e: Status 404 returned error can't find the container with id 7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e
Nov 03 07:52:58.109542 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:58.109527    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:58.112390 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:58.112376    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:52:58.119892 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:58.119871    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:52:58.120741 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:58.120726    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:52:58.135245 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:58.135225    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:52:58.171271 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:52:58.171246    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f16205502b8d2684b7e492677380c9.slice/crio-e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1 WatchSource:0}: Error finding container e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1: Status 404 returned error can't find the container with id e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1
Nov 03 07:52:59.949541 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.949514    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:59.969580 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.969554    2574 apiserver.go:52] "Watching apiserver"
Nov 03 07:52:59.980512 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.980488    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 07:52:59.980877 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.980854    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-cknqz","openshift-network-diagnostics/network-check-target-k5mr4","openshift-ovn-kubernetes/ovnkube-node-dgl7z","kube-system/global-pull-secret-syncer-898qp","kube-system/konnectivity-agent-254xf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn","openshift-cluster-node-tuning-operator/tuned-f5b6x","openshift-dns/node-resolver-j7zvx","openshift-image-registry/node-ca-srvsw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal","openshift-multus/multus-f6vt5","kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-6tqbr","openshift-multus/network-metrics-daemon-m7nbf"]
Nov 03 07:52:59.983500 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.983474    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:52:59.985927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.985907    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:52:59.987939 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.987909    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:52:59.988041 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.987981    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 07:52:59.988126 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.988084    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:52:59.988585 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.988196    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpwxk\""
Nov 03 07:52:59.989507 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.989487    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 07:52:59.990090 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.990049    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 07:52:59.990183 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.990175    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c2t62\""
Nov 03 07:52:59.990290 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.990213    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 07:52:59.990290 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.990251    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 07:52:59.990387 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.990291    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 07:52:59.990387 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.990368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:52:59.990488 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:52:59.990439    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:52:59.991081 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.991050    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 07:52:59.991853 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.991764    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 07:52:59.992289 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.992022    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 07:52:59.992289 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.992124    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 07:52:59.992289 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.992029    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 07:52:59.992289 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.992219    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 07:52:59.992531 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.992289    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9glt\""
Nov 03 07:52:59.992531 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.992418    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:52:59.993905 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.993889    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 07:52:59.995475 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.994833    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:52:59.997530 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:52:59.997509    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.000109 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.000091    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.002561 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-systemd-units\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.002561 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-etc-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.002561 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-node-log\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.002561 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-env-overrides\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ecea087-8175-4789-a497-7beeb69e1610-agent-certs\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002658    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-config\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7hdg\" (UniqueName: \"kubernetes.io/projected/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-kube-api-access-n7hdg\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-dbus\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.002788 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-systemd\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-netd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-kubernetes\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-lib-modules\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-tuned\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-slash\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-var-lib-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-ovn\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.002992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ecea087-8175-4789-a497-7beeb69e1610-konnectivity-ca\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwwk\" (UniqueName: \"kubernetes.io/projected/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-kube-api-access-2fwwk\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-cnibin\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-netns\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-bin\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-script-lib\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-var-lib-kubelet\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-tmp\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003260    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-kubelet\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003310    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-systemd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysconfig\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-sys\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-kubelet-config\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-modprobe-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.003458 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-host\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.004218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovn-node-metrics-cert\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.004218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-system-cni-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.004218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003515    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.004218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9gls\" (UniqueName: \"kubernetes.io/projected/626fe428-7557-42b8-a473-0ab960c59898-kube-api-access-d9gls\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.004218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-log-socket\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.004218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-conf\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.004218 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.003567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-run\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.004894 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.004876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:00.004985 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.004943    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:00.007845 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.007827    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:00.007943 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.007875    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 07:53:00.007943 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.007889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:00.009871 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.009852    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.013556 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.013537    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.021143 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.021123    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g57t6"
Nov 03 07:53:00.034448 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.034426    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g57t6"
Nov 03 07:53:00.088973 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.088941    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 07:53:00.089151 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.089041    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 07:53:00.089209 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.089196    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dc6md\""
Nov 03 07:53:00.089280 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.089266    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:00.089332 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.089293    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 07:53:00.089372 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.089349    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2mmmx\""
Nov 03 07:53:00.090912 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.090893    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 07:53:00.091028 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.090913    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 07:53:00.091028 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.090930    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 07:53:00.091209 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091076    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 07:53:00.091209 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091099    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:00.091209 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091143    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 07:53:00.091209 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091165    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4xzk\""
Nov 03 07:53:00.091209 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091207    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 07:53:00.091471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091240    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 07:53:00.091471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091311    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 07:53:00.091471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091351    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-thqm7\""
Nov 03 07:53:00.091471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091364    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwf4\""
Nov 03 07:53:00.091471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091394    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 07:53:00.091471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091422    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bmmvq\""
Nov 03 07:53:00.091771 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.091753    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 07:53:00.103825 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.103799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-hosts-file\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.103922 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.103839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-cnibin\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.103922 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.103912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-netns\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.103937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.103959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-var-lib-kubelet\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.103973    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-cnibin\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.104027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.103984    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-cni-binary-copy\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.104219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104025    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-netns\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfg5\" (UniqueName: \"kubernetes.io/projected/f0eca957-f922-4106-8192-4d3d39e8a21d-kube-api-access-hgfg5\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.104219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104113    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.104219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104172    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-var-lib-kubelet\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-systemd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysconfig\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-registration-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104280    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-netns\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-kubelet-config\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104331    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-systemd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-modprobe-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104373    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-kubelet-config\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104416    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysconfig\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104477 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-system-cni-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104508    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-system-cni-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gls\" (UniqueName: \"kubernetes.io/projected/626fe428-7557-42b8-a473-0ab960c59898-kube-api-access-d9gls\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104566    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-modprobe-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104591    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-run\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104625    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-node-log\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104757    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-run\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ecea087-8175-4789-a497-7beeb69e1610-agent-certs\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104771    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-node-log\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-systemd\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-bin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6467\" (UniqueName: \"kubernetes.io/projected/a533b40e-1b38-48c6-b409-d12ead6930a0-kube-api-access-w6467\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104894    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-systemd\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-tmp-dir\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.104988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.104979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7hdg\" (UniqueName: \"kubernetes.io/projected/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-kube-api-access-n7hdg\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-dbus\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105097    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105393    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf46t\" (UniqueName: \"kubernetes.io/projected/26fe52bf-f29e-465a-b519-164d0f092c0b-kube-api-access-nf46t\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-system-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105475    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-hostroot\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-conf-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105561    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-netd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-lib-modules\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105627    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-dbus\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-tuned\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-cnibin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105668    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-netd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.105709 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-multus-certs\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105782    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-lib-modules\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5gg7\" (UniqueName: \"kubernetes.io/projected/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-kube-api-access-b5gg7\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0eca957-f922-4106-8192-4d3d39e8a21d-host-slash\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-ovn\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ecea087-8175-4789-a497-7beeb69e1610-konnectivity-ca\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-bin\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-ovn\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.105988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-script-lib\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106024    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-bin\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-tmp\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-socket-dir-parent\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-multus\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-daemon-config\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-serviceca\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-kubelet\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.106506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-sys\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106427    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-k8s-cni-cncf-io\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-kubelet\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-etc-kubernetes\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-host\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-sys-fs\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlkx\" (UniqueName: \"kubernetes.io/projected/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-kube-api-access-5xlkx\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovn-node-metrics-cert\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106598    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-script-lib\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-log-socket\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-conf\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106643    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106794    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-conf\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-host\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-systemd-units\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107201 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106882    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ecea087-8175-4789-a497-7beeb69e1610-konnectivity-ca\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106931    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-etc-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-etc-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.106967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-env-overrides\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107005    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-device-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-config\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0eca957-f922-4106-8192-4d3d39e8a21d-iptables-alerter-script\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkd4v\" (UniqueName: \"kubernetes.io/projected/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-kube-api-access-vkd4v\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-kubernetes\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-socket-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-slash\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107346    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-var-lib-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107399    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-kubelet\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.107673 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107442    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-sys\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.108258 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.108258 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwwk\" (UniqueName: \"kubernetes.io/projected/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-kube-api-access-2fwwk\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.108258 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107497    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:00.108258 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107497    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-kubernetes\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.108258 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.107601    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-systemd-units\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.108258 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.108144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-slash\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.108258 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.108192    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-var-lib-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.108956 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.108238    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-log-socket\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.108956 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.108546    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-host\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.108956 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.108632    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:00.108956 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.108693    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret podName:342d6281-4bdd-4090-ac8b-717ca2942a0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:00.608666761 +0000 UTC m=+2.967711455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret") pod "global-pull-secret-syncer-898qp" (UID: "342d6281-4bdd-4090-ac8b-717ca2942a0a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:00.111251 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.109178    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-env-overrides\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.111251 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.109515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-config\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.111251 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.109650    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.111251 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.110716    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-tmp\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.111618 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.111550    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-tuned\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.111957 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.111915    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ecea087-8175-4789-a497-7beeb69e1610-agent-certs\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:00.113727 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.113685    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovn-node-metrics-cert\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.115262 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.115200    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gls\" (UniqueName: \"kubernetes.io/projected/626fe428-7557-42b8-a473-0ab960c59898-kube-api-access-d9gls\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.116014 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.115992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7hdg\" (UniqueName: \"kubernetes.io/projected/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-kube-api-access-n7hdg\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.117046 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.116803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwwk\" (UniqueName: \"kubernetes.io/projected/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-kube-api-access-2fwwk\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.145019 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.144976    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"40f16205502b8d2684b7e492677380c9","Type":"ContainerStarted","Data":"e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1"}
Nov 03 07:53:00.146298 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.146274    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"a15f1c78c19d0b39887f68042df09ebe","Type":"ContainerStarted","Data":"7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e"}
Nov 03 07:53:00.208234 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:00.208234 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-hosts-file\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.208234 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-cni-binary-copy\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.208234 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfg5\" (UniqueName: \"kubernetes.io/projected/f0eca957-f922-4106-8192-4d3d39e8a21d-kube-api-access-hgfg5\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208332    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-hosts-file\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-registration-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208415    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-registration-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-netns\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-bin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6467\" (UniqueName: \"kubernetes.io/projected/a533b40e-1b38-48c6-b409-d12ead6930a0-kube-api-access-w6467\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208502    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-netns\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208517    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.208546 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-tmp-dir\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf46t\" (UniqueName: \"kubernetes.io/projected/26fe52bf-f29e-465a-b519-164d0f092c0b-kube-api-access-nf46t\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208557    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-bin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-system-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-hostroot\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-conf-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208728    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-system-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-cnibin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-multus-certs\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208768    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-cni-binary-copy\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208772    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-hostroot\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5gg7\" (UniqueName: \"kubernetes.io/projected/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-kube-api-access-b5gg7\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208808    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-tmp-dir\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-conf-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0eca957-f922-4106-8192-4d3d39e8a21d-host-slash\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208815    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-cnibin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208834    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-multus-certs\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208851    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0eca957-f922-4106-8192-4d3d39e8a21d-host-slash\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.209015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208963    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-socket-dir-parent\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.208987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-multus\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-daemon-config\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-serviceca\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-k8s-cni-cncf-io\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-kubelet\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-etc-kubernetes\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-sys-fs\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209113    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-multus\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209176    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-kubelet\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlkx\" (UniqueName: \"kubernetes.io/projected/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-kube-api-access-5xlkx\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209191    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-socket-dir-parent\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209218    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-k8s-cni-cncf-io\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209242    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-sys-fs\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-host\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.209245    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:00.209855 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-device-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209279    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-host\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209316    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-device-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0eca957-f922-4106-8192-4d3d39e8a21d-iptables-alerter-script\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209342    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-etc-kubernetes\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.209345    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs podName:6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf nodeName:}" failed. No retries permitted until 2026-11-03 07:53:00.70932333 +0000 UTC m=+3.068368024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs") pod "network-metrics-daemon-m7nbf" (UID: "6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkd4v\" (UniqueName: \"kubernetes.io/projected/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-kube-api-access-vkd4v\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-socket-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209562    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-socket-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209613    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-daemon-config\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209714    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209732    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-serviceca\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.210402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.209846    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0eca957-f922-4106-8192-4d3d39e8a21d-iptables-alerter-script\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.215003 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.214983    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:53:00.215003 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.215005    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:53:00.215182 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.215017    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qq5w for pod openshift-network-diagnostics/network-check-target-k5mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:00.215182 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.215087    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w podName:76974834-71dc-4488-b393-ed2730c0276a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:00.715055385 +0000 UTC m=+3.074100089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5qq5w" (UniqueName: "kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w") pod "network-check-target-k5mr4" (UID: "76974834-71dc-4488-b393-ed2730c0276a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:00.217877 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.217853    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5gg7\" (UniqueName: \"kubernetes.io/projected/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-kube-api-access-b5gg7\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.218040 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.218019    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6467\" (UniqueName: \"kubernetes.io/projected/a533b40e-1b38-48c6-b409-d12ead6930a0-kube-api-access-w6467\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.218316 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.218293    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfg5\" (UniqueName: \"kubernetes.io/projected/f0eca957-f922-4106-8192-4d3d39e8a21d-kube-api-access-hgfg5\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.218703 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.218683    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkd4v\" (UniqueName: \"kubernetes.io/projected/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-kube-api-access-vkd4v\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.219249 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.219218    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlkx\" (UniqueName: \"kubernetes.io/projected/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-kube-api-access-5xlkx\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:00.219465 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.219448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf46t\" (UniqueName: \"kubernetes.io/projected/26fe52bf-f29e-465a-b519-164d0f092c0b-kube-api-access-nf46t\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.278750 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.278718    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:53:00.299164 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.299134    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:53:00.310161 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.310134    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426baa2b_1ff2_4e15_bf7d_4efdcdf4b221.slice/crio-7e90f00a294ad5f41e2c4f7c0edd143e29a21a44f9b13d9a8541c480a6f26cbe WatchSource:0}: Error finding container 7e90f00a294ad5f41e2c4f7c0edd143e29a21a44f9b13d9a8541c480a6f26cbe: Status 404 returned error can't find the container with id 7e90f00a294ad5f41e2c4f7c0edd143e29a21a44f9b13d9a8541c480a6f26cbe
Nov 03 07:53:00.329707 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.329677    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:53:00.335165 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.335141    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:00.340135 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.340105    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626fe428_7557_42b8_a473_0ab960c59898.slice/crio-079f77c5974971199aa139b9a43960bf5a69d77e69deebb7c4d26f5aab34a998 WatchSource:0}: Error finding container 079f77c5974971199aa139b9a43960bf5a69d77e69deebb7c4d26f5aab34a998: Status 404 returned error can't find the container with id 079f77c5974971199aa139b9a43960bf5a69d77e69deebb7c4d26f5aab34a998
Nov 03 07:53:00.345667 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.345641    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecea087_8175_4789_a497_7beeb69e1610.slice/crio-46aeb6b2f93ef9c35817bf1c47f4294779f5b0719b7688c0d67a6f6a10f255ec WatchSource:0}: Error finding container 46aeb6b2f93ef9c35817bf1c47f4294779f5b0719b7688c0d67a6f6a10f255ec: Status 404 returned error can't find the container with id 46aeb6b2f93ef9c35817bf1c47f4294779f5b0719b7688c0d67a6f6a10f255ec
Nov 03 07:53:00.350867 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.350707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:53:00.352924 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.352883    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:53:00.354174 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.354036    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:00.356447 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.356426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:53:00.361756 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.361677    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18c61c8_db8c_49ca_8fbd_9378be7e9c51.slice/crio-73d983f21fc0e06c47c090cd39c024b8aa55e597bcc940eb514518a761a58b66 WatchSource:0}: Error finding container 73d983f21fc0e06c47c090cd39c024b8aa55e597bcc940eb514518a761a58b66: Status 404 returned error can't find the container with id 73d983f21fc0e06c47c090cd39c024b8aa55e597bcc940eb514518a761a58b66
Nov 03 07:53:00.365006 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.364956    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fe52bf_f29e_465a_b519_164d0f092c0b.slice/crio-114512a799f2d7eb864d83a4aff0eefb65c2e5c3961aa714de039cc981fc883f WatchSource:0}: Error finding container 114512a799f2d7eb864d83a4aff0eefb65c2e5c3961aa714de039cc981fc883f: Status 404 returned error can't find the container with id 114512a799f2d7eb864d83a4aff0eefb65c2e5c3961aa714de039cc981fc883f
Nov 03 07:53:00.368290 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.368273    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6vt5"
Nov 03 07:53:00.370310 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.370289    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab85a3f2_5d53_4d6a_bf1b_185db437abf1.slice/crio-42904b61dc9a39cabb621a14bbc9d5cba600731562df9d4d2db41f2f6c3376e6 WatchSource:0}: Error finding container 42904b61dc9a39cabb621a14bbc9d5cba600731562df9d4d2db41f2f6c3376e6: Status 404 returned error can't find the container with id 42904b61dc9a39cabb621a14bbc9d5cba600731562df9d4d2db41f2f6c3376e6
Nov 03 07:53:00.372714 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.372674    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0eca957_f922_4106_8192_4d3d39e8a21d.slice/crio-3592033c3df3e86957ca9685c0078fcb41b78dbbf448ac9a0c6c339d66ebdb6f WatchSource:0}: Error finding container 3592033c3df3e86957ca9685c0078fcb41b78dbbf448ac9a0c6c339d66ebdb6f: Status 404 returned error can't find the container with id 3592033c3df3e86957ca9685c0078fcb41b78dbbf448ac9a0c6c339d66ebdb6f
Nov 03 07:53:00.373828 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.373665    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:53:00.379591 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.379561    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda533b40e_1b38_48c6_b409_d12ead6930a0.slice/crio-23de0469b8e597085dccb0d1c2b1d399002ff65d13c757f69ff267581253ae88 WatchSource:0}: Error finding container 23de0469b8e597085dccb0d1c2b1d399002ff65d13c757f69ff267581253ae88: Status 404 returned error can't find the container with id 23de0469b8e597085dccb0d1c2b1d399002ff65d13c757f69ff267581253ae88
Nov 03 07:53:00.386164 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:00.386144    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c1f0f2_b9b6_4659_97fb_9b3092e04f5f.slice/crio-b31983de7934c38ad45951a23fdc243078200f7a6041a8464a46fbd921bc6713 WatchSource:0}: Error finding container b31983de7934c38ad45951a23fdc243078200f7a6041a8464a46fbd921bc6713: Status 404 returned error can't find the container with id b31983de7934c38ad45951a23fdc243078200f7a6041a8464a46fbd921bc6713
Nov 03 07:53:00.612026 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.611957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:00.612174 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.612125    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:00.612246 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.612185    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret podName:342d6281-4bdd-4090-ac8b-717ca2942a0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:01.612166136 +0000 UTC m=+3.971210817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret") pod "global-pull-secret-syncer-898qp" (UID: "342d6281-4bdd-4090-ac8b-717ca2942a0a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:00.712646 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.712611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:00.712795 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.712776    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:00.712863 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.712832    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs podName:6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf nodeName:}" failed. No retries permitted until 2026-11-03 07:53:01.71281562 +0000 UTC m=+4.071860316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs") pod "network-metrics-daemon-m7nbf" (UID: "6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:00.814040 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:00.814010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:00.814221 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.814199    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:53:00.814280 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.814225    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:53:00.814280 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.814238    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qq5w for pod openshift-network-diagnostics/network-check-target-k5mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:00.814373 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:00.814301    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w podName:76974834-71dc-4488-b393-ed2730c0276a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:01.814281931 +0000 UTC m=+4.173326632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qq5w" (UniqueName: "kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w") pod "network-check-target-k5mr4" (UID: "76974834-71dc-4488-b393-ed2730c0276a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:01.038148 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.038017    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:48:00 +0000 UTC" deadline="2028-07-18 05:51:14.054388462 +0000 UTC"
Nov 03 07:53:01.038148 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.038088    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14949h58m13.016304733s"
Nov 03 07:53:01.169590 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.168771    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"40f16205502b8d2684b7e492677380c9","Type":"ContainerStarted","Data":"5eb743171082d7f75f0f428382796a40fc0c63fa860d6911e19f603296d007b5"}
Nov 03 07:53:01.170982 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.170930    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6vt5" event={"ID":"a533b40e-1b38-48c6-b409-d12ead6930a0","Type":"ContainerStarted","Data":"23de0469b8e597085dccb0d1c2b1d399002ff65d13c757f69ff267581253ae88"}
Nov 03 07:53:01.189438 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.189338    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn" event={"ID":"26fe52bf-f29e-465a-b519-164d0f092c0b","Type":"ContainerStarted","Data":"114512a799f2d7eb864d83a4aff0eefb65c2e5c3961aa714de039cc981fc883f"}
Nov 03 07:53:01.190955 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.190922    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-254xf" event={"ID":"4ecea087-8175-4789-a497-7beeb69e1610","Type":"ContainerStarted","Data":"46aeb6b2f93ef9c35817bf1c47f4294779f5b0719b7688c0d67a6f6a10f255ec"}
Nov 03 07:53:01.195457 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.195427    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5b6x" event={"ID":"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221","Type":"ContainerStarted","Data":"7e90f00a294ad5f41e2c4f7c0edd143e29a21a44f9b13d9a8541c480a6f26cbe"}
Nov 03 07:53:01.199684 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.199620    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7zvx" event={"ID":"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f","Type":"ContainerStarted","Data":"b31983de7934c38ad45951a23fdc243078200f7a6041a8464a46fbd921bc6713"}
Nov 03 07:53:01.208778 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.208753    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cknqz" event={"ID":"f0eca957-f922-4106-8192-4d3d39e8a21d","Type":"ContainerStarted","Data":"3592033c3df3e86957ca9685c0078fcb41b78dbbf448ac9a0c6c339d66ebdb6f"}
Nov 03 07:53:01.221625 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.221598    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"42904b61dc9a39cabb621a14bbc9d5cba600731562df9d4d2db41f2f6c3376e6"}
Nov 03 07:53:01.223816 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.223785    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-srvsw" event={"ID":"e18c61c8-db8c-49ca-8fbd-9378be7e9c51","Type":"ContainerStarted","Data":"73d983f21fc0e06c47c090cd39c024b8aa55e597bcc940eb514518a761a58b66"}
Nov 03 07:53:01.227320 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.227296    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerStarted","Data":"079f77c5974971199aa139b9a43960bf5a69d77e69deebb7c4d26f5aab34a998"}
Nov 03 07:53:01.267082 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.267032    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:53:01.625463 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.624880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:01.625463 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.625027    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:01.625463 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.625107    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret podName:342d6281-4bdd-4090-ac8b-717ca2942a0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:03.625089544 +0000 UTC m=+5.984134231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret") pod "global-pull-secret-syncer-898qp" (UID: "342d6281-4bdd-4090-ac8b-717ca2942a0a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:01.726120 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.725859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:01.726120 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.726003    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:01.726755 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.726726    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs podName:6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf nodeName:}" failed. No retries permitted until 2026-11-03 07:53:03.726047244 +0000 UTC m=+6.085091940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs") pod "network-metrics-daemon-m7nbf" (UID: "6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:01.827398 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:01.826703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:01.827398 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.826887    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:53:01.827398 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.826911    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:53:01.827398 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.826924    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qq5w for pod openshift-network-diagnostics/network-check-target-k5mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:01.827398 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:01.826978    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w podName:76974834-71dc-4488-b393-ed2730c0276a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:03.826962303 +0000 UTC m=+6.186006997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qq5w" (UniqueName: "kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w") pod "network-check-target-k5mr4" (UID: "76974834-71dc-4488-b393-ed2730c0276a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:02.039764 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.039655    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:48:00 +0000 UTC" deadline="2028-07-09 00:26:27.770003436 +0000 UTC"
Nov 03 07:53:02.039764 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.039714    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14728h33m25.730293242s"
Nov 03 07:53:02.143653 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.143622    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:02.143823 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:02.143749    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:02.143823 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.143809    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:02.143937 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:02.143872    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:02.143937 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.143915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:02.144134 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:02.143994    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:02.233081 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.233033    2574 generic.go:356] "Generic (PLEG): container finished" podID="a15f1c78c19d0b39887f68042df09ebe" containerID="fe21e5fd76dbbbc045d498cc53a21f0ef9441bdffe61119fcef9ca48d0a6884e" exitCode=0
Nov 03 07:53:02.234088 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.234046    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"a15f1c78c19d0b39887f68042df09ebe","Type":"ContainerDied","Data":"fe21e5fd76dbbbc045d498cc53a21f0ef9441bdffe61119fcef9ca48d0a6884e"}
Nov 03 07:53:02.249623 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:02.249582    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal" podStartSLOduration=3.249567936 podStartE2EDuration="3.249567936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:01.187113761 +0000 UTC m=+3.546158464" watchObservedRunningTime="2026-11-03 07:53:02.249567936 +0000 UTC m=+4.608612638"
Nov 03 07:53:03.241850 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:03.241798    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"a15f1c78c19d0b39887f68042df09ebe","Type":"ContainerStarted","Data":"91c435a3d5b3333b6ef406823f9dc8d56b28ff42bb88936102441034d84bf0de"}
Nov 03 07:53:03.641551 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:03.640941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:03.641551 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.641128    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:03.641551 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.641187    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret podName:342d6281-4bdd-4090-ac8b-717ca2942a0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:07.641170015 +0000 UTC m=+10.000214711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret") pod "global-pull-secret-syncer-898qp" (UID: "342d6281-4bdd-4090-ac8b-717ca2942a0a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:03.741390 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:03.741347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:03.741559 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.741542    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:03.741616 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.741603    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs podName:6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf nodeName:}" failed. No retries permitted until 2026-11-03 07:53:07.741585988 +0000 UTC m=+10.100630687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs") pod "network-metrics-daemon-m7nbf" (UID: "6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:03.842585 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:03.842508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:03.842743 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.842674    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:53:03.842743 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.842693    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:53:03.842743 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.842704    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qq5w for pod openshift-network-diagnostics/network-check-target-k5mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:03.842887 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:03.842754    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w podName:76974834-71dc-4488-b393-ed2730c0276a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:07.842737387 +0000 UTC m=+10.201782082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qq5w" (UniqueName: "kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w") pod "network-check-target-k5mr4" (UID: "76974834-71dc-4488-b393-ed2730c0276a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:04.143464 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:04.142735    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:04.143464 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:04.142857    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:04.143464 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:04.143228    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:04.143464 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:04.143308    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:04.143464 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:04.143353    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:04.143464 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:04.143400    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:04.260395 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:04.259165    2574 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="4da97e28241da3da8e95f5447dece0428ff4d9803022c0fd3e92e66f256dde76" exitCode=0
Nov 03 07:53:04.260395 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:04.260183    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerDied","Data":"4da97e28241da3da8e95f5447dece0428ff4d9803022c0fd3e92e66f256dde76"}
Nov 03 07:53:04.293079 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:04.287590    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal" podStartSLOduration=5.28756353 podStartE2EDuration="5.28756353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:03.260411909 +0000 UTC m=+5.619456612" watchObservedRunningTime="2026-11-03 07:53:04.28756353 +0000 UTC m=+6.646608238"
Nov 03 07:53:05.264481 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:05.264435    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-254xf" event={"ID":"4ecea087-8175-4789-a497-7beeb69e1610","Type":"ContainerStarted","Data":"262e54346fc155097fc19a772059839261acc46a9922848aca2245feb0ad7c4b"}
Nov 03 07:53:05.281718 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:05.281660    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-254xf" podStartSLOduration=3.382035939 podStartE2EDuration="7.281642278s" totalImagesPullingTime="3.899606339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:05.2812037 +0000 UTC m=+7.640248406" watchObservedRunningTime="2026-11-03 07:53:05.281642278 +0000 UTC m=+7.640686998"
Nov 03 07:53:06.140644 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:06.140618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:06.140740 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:06.140645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:06.140740 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:06.140714    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:06.140839 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:06.140791    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:06.140942 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:06.140848    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:06.140942 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:06.140912    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:06.269613 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:06.269573    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-srvsw" event={"ID":"e18c61c8-db8c-49ca-8fbd-9378be7e9c51","Type":"ContainerStarted","Data":"8bd18fc95b8f143c1664b11b5b821c65013d1c3688b02ba346e54adb5864ce77"}
Nov 03 07:53:06.272815 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:06.272787    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn" event={"ID":"26fe52bf-f29e-465a-b519-164d0f092c0b","Type":"ContainerStarted","Data":"53f7c8ae02813411c7f21a43faeddfc22522837ad84d7386baa80a10acc3d854"}
Nov 03 07:53:07.673443 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:07.672735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:07.673443 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.672956    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:07.673443 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.673023    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret podName:342d6281-4bdd-4090-ac8b-717ca2942a0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:15.673003042 +0000 UTC m=+18.032047722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret") pod "global-pull-secret-syncer-898qp" (UID: "342d6281-4bdd-4090-ac8b-717ca2942a0a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:07.773159 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:07.773113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:07.773355 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.773336    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:07.773428 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.773408    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs podName:6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf nodeName:}" failed. No retries permitted until 2026-11-03 07:53:15.77338954 +0000 UTC m=+18.132434242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs") pod "network-metrics-daemon-m7nbf" (UID: "6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:07.867511 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:07.867303    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 07:53:07.874345 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:07.874311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:07.874563 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.874547    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:53:07.874619 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.874570    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:53:07.874619 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.874583    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qq5w for pod openshift-network-diagnostics/network-check-target-k5mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:07.874710 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:07.874653    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w podName:76974834-71dc-4488-b393-ed2730c0276a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:15.874634414 +0000 UTC m=+18.233679111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qq5w" (UniqueName: "kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w") pod "network-check-target-k5mr4" (UID: "76974834-71dc-4488-b393-ed2730c0276a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:08.054678 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.053876    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T07:53:07.867323233Z","UUID":"bcb3220d-8cf9-4b90-959c-fb505fc539b0","Handler":null,"Name":"","Endpoint":""}
Nov 03 07:53:08.061506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.061227    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 07:53:08.061506 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.061261    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 07:53:08.144223 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.144187    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:08.144377 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:08.144345    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:08.144453 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.144431    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:08.144542 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:08.144508    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:08.144679 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.144646    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:08.144736 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:08.144711    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:08.184432 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.184405    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:08.185175 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.185151    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:08.202966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.202910    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-srvsw" podStartSLOduration=5.265614156 podStartE2EDuration="10.202888954s" totalImagesPullingTime="4.937274798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:06.286704826 +0000 UTC m=+8.645749532" watchObservedRunningTime="2026-11-03 07:53:08.202888954 +0000 UTC m=+10.561933686"
Nov 03 07:53:08.281420 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.281383    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6vt5" event={"ID":"a533b40e-1b38-48c6-b409-d12ead6930a0","Type":"ContainerStarted","Data":"030b0f08d4eff5a440c7958c16ddead9014707b91b8ff0ae7611ea595b7db523"}
Nov 03 07:53:08.284544 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.284509    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn" event={"ID":"26fe52bf-f29e-465a-b519-164d0f092c0b","Type":"ContainerStarted","Data":"80c0b05f89a2a5cbc6fb2cb2ff6cf5156226b2ef87cbcb77f195eb3890184de5"}
Nov 03 07:53:08.284717 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.284580    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:08.284988 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.284971    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:53:08.300604 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:08.300553    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-f6vt5" podStartSLOduration=2.506781942 podStartE2EDuration="10.300533434s" totalImagesPullingTime="7.793751492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:08.300167121 +0000 UTC m=+10.659211825" watchObservedRunningTime="2026-11-03 07:53:08.300533434 +0000 UTC m=+10.659578137"
Nov 03 07:53:09.287536 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:09.287305    2574 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="3c1ec5884f86d8b3fb4adba0c1bb4d68021477546f0508106609ab39f5897f5f" exitCode=0
Nov 03 07:53:09.287536 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:09.287386    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerDied","Data":"3c1ec5884f86d8b3fb4adba0c1bb4d68021477546f0508106609ab39f5897f5f"}
Nov 03 07:53:09.289800 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:09.289759    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn" event={"ID":"26fe52bf-f29e-465a-b519-164d0f092c0b","Type":"ContainerStarted","Data":"d4ddef9f53f44cdf7c7daa92ae5d32cb3d2dd212bd76206831001011e233334d"}
Nov 03 07:53:09.291131 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:09.291107    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7zvx" event={"ID":"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f","Type":"ContainerStarted","Data":"1d9920a8605330edbcfb0e35fbe69f7fc134dab5cb6eebd8ca4f00015e21cae4"}
Nov 03 07:53:09.343436 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:09.343381    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn" podStartSLOduration=3.149343691 podStartE2EDuration="11.343363423s" totalImagesPullingTime="8.194019732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:09.327366822 +0000 UTC m=+11.686411526" watchObservedRunningTime="2026-11-03 07:53:09.343363423 +0000 UTC m=+11.702408129"
Nov 03 07:53:10.026177 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.026145    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:10.139832 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.139751    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:10.139953 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.139854    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:10.139986 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.139963    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:10.140021 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:10.139987    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:10.140083 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:10.140026    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:10.140138 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:10.140114    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:10.209869 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.209839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:10.294601 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.294565    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cknqz" event={"ID":"f0eca957-f922-4106-8192-4d3d39e8a21d","Type":"ContainerStarted","Data":"2648fd7d27dde656d2b166860b4e022646f10ce57880fb55cc8b4ad72ecd5a55"}
Nov 03 07:53:10.311317 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.311266    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cknqz" podStartSLOduration=4.391709387 podStartE2EDuration="12.311247847s" totalImagesPullingTime="7.91953846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:10.310982819 +0000 UTC m=+12.670027522" watchObservedRunningTime="2026-11-03 07:53:10.311247847 +0000 UTC m=+12.670292552"
Nov 03 07:53:10.311884 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.311853    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-j7zvx" podStartSLOduration=4.263292633 podStartE2EDuration="12.311845263s" totalImagesPullingTime="8.04855263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:09.342834907 +0000 UTC m=+11.701879609" watchObservedRunningTime="2026-11-03 07:53:10.311845263 +0000 UTC m=+12.670889969"
Nov 03 07:53:10.885951 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:10.885846    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:11.298486 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:11.298398    2574 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="d6acda9d6ab18c438a5c460c5d50f6b0075e50174e3ebc30b02097055b4538e8" exitCode=0
Nov 03 07:53:11.298486 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:11.298433    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerDied","Data":"d6acda9d6ab18c438a5c460c5d50f6b0075e50174e3ebc30b02097055b4538e8"}
Nov 03 07:53:11.300365 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:11.300330    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5b6x" event={"ID":"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221","Type":"ContainerStarted","Data":"4ea8ae6f1ee07e6b48fa0fee581cea07125fcdd3aeb7b52cd09375c2db14dda0"}
Nov 03 07:53:11.342604 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:11.342562    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f5b6x" podStartSLOduration=3.149644175 podStartE2EDuration="13.342551282s" totalImagesPullingTime="10.192907107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:11.342385686 +0000 UTC m=+13.701430387" watchObservedRunningTime="2026-11-03 07:53:11.342551282 +0000 UTC m=+13.701595983"
Nov 03 07:53:11.891216 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:11.891188    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:11.996834 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:11.996806    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:12.140561 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:12.140538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:12.140561 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:12.140550    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:12.140701 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:12.140610    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:12.140772 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:12.140732    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:12.140921 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:12.140897    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:12.141073 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:12.141028    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:12.304840 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:12.304766    2574 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="f6cc3519587c7f62a38ce7e491eef0f177888e9630e584d9e2d8a5f9f5c59946" exitCode=0
Nov 03 07:53:12.305326 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:12.304843    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerDied","Data":"f6cc3519587c7f62a38ce7e491eef0f177888e9630e584d9e2d8a5f9f5c59946"}
Nov 03 07:53:14.117036 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:14.116833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:14.140216 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:14.140174    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:14.140383 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:14.140239    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:14.140383 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:14.140241    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:14.140383 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:14.140321    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:14.140552 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:14.140461    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:14.140604 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:14.140550    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:14.220357 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:14.220322    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:15.315351 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:15.315315    2574 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="427746251aa2f82f883ca20911cdee35513ed1388e34f320bcca3412fdeb00e8" exitCode=0
Nov 03 07:53:15.315814 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:15.315371    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerDied","Data":"427746251aa2f82f883ca20911cdee35513ed1388e34f320bcca3412fdeb00e8"}
Nov 03 07:53:15.615558 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:15.615478    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:15.724682 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:15.724653    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:15.730714 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:15.730691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:15.730830 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.730815    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:15.730883 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.730868    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret podName:342d6281-4bdd-4090-ac8b-717ca2942a0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:31.730856193 +0000 UTC m=+34.089900872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret") pod "global-pull-secret-syncer-898qp" (UID: "342d6281-4bdd-4090-ac8b-717ca2942a0a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:15.831471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:15.831435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:15.831629 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.831596    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:15.831681 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.831657    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs podName:6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf nodeName:}" failed. No retries permitted until 2026-11-03 07:53:31.831641591 +0000 UTC m=+34.190686272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs") pod "network-metrics-daemon-m7nbf" (UID: "6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:15.932008 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:15.931938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:15.932169 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.932129    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:53:15.932169 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.932156    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:53:15.932169 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.932168    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qq5w for pod openshift-network-diagnostics/network-check-target-k5mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:15.932326 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:15.932224    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w podName:76974834-71dc-4488-b393-ed2730c0276a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:31.932210626 +0000 UTC m=+34.291255305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qq5w" (UniqueName: "kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w") pod "network-check-target-k5mr4" (UID: "76974834-71dc-4488-b393-ed2730c0276a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:16.140262 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.140227    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:16.140424 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.140235    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:16.140424 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:16.140368    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:16.140424 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.140235    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:16.140576 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:16.140427    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:16.140576 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:16.140507    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:16.312013 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.311949    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:16.320277 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.320253    2574 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="05483ec690fd98782493b215fefb1387db43a011e3c7dbd9c5813802a448b15c" exitCode=0
Nov 03 07:53:16.320605 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.320291    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerDied","Data":"05483ec690fd98782493b215fefb1387db43a011e3c7dbd9c5813802a448b15c"}
Nov 03 07:53:16.423628 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.423594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:16.535247 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.535050    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:16.829210 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.829145    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:16.939954 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:16.939926    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:17.324839 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.324737    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"43365c5e75a18ec9c18d453f22c6dc4f02f7b684c3937ec86bfbaac92facef30"}
Nov 03 07:53:17.325314 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.324848    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"e661cc23f3ec84ba248ad192bb5414765586385f6d6325c0f3e07408936e9012"}
Nov 03 07:53:17.325314 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.324894    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"09e016fe255e897c66ae06a73b97abe27c7cb3780793f3b4bfef33946e23729b"}
Nov 03 07:53:17.325314 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.324938    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"1bee204297d433d90ce01a47c300206f08e08bb556d83ef031543bca7925a3ed"}
Nov 03 07:53:17.325314 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.324984    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"9e949791d415c8c840ff9b1acf7eb898b9c4a185d6d99e383d7dc18c256fe085"}
Nov 03 07:53:17.325314 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.325030    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"a6f7bbf3b907088c21f683277df78ceaa036bdb81ea81d6c2160592cc910f095"}
Nov 03 07:53:17.327879 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.327855    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" event={"ID":"626fe428-7557-42b8-a473-0ab960c59898","Type":"ContainerStarted","Data":"936c294311144a9d88a25bef38b90cc5b86c9f03c5759f23d33d91863af8b23e"}
Nov 03 07:53:17.357989 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.357954    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6tqbr" podStartSLOduration=8.397820155 podStartE2EDuration="19.357941644s" totalImagesPullingTime="10.960121489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:17.35536188 +0000 UTC m=+19.714406582" watchObservedRunningTime="2026-11-03 07:53:17.357941644 +0000 UTC m=+19.716986369"
Nov 03 07:53:17.410660 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.410641    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:17.536868 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.536844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:17.995161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:17.995134    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:18.141323 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:18.141271    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:18.141560 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:18.141311    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:18.141680 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:18.141469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:18.141730 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:18.141686    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:18.144125 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:18.142231    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:18.144125 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:18.142357    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:18.174721 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:18.174695    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:18.299032 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:18.298968    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:18.749815 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:18.749780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:19.334215 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:19.334178    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"a2c7f7a4447b4bb70947948e76f626a9c0e1efd03b7e41d2900d78c1f2d9e40d"}
Nov 03 07:53:19.549460 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:19.549428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:19.742203 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:19.742139    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:20.139642 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:20.139552    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:20.139642 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:20.139640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:20.140173 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:20.139670    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:20.140173 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:20.139739    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:20.140173 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:20.139799    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:20.140173 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:20.139848    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:20.140173 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:20.139913    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:20.340307 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:20.340274    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:20.952987 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:20.952958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:21.341191 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.341153    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" event={"ID":"ab85a3f2-5d53-4d6a-bf1b-185db437abf1","Type":"ContainerStarted","Data":"c305a7fa4a5ad143cb4d907acc8b828ed1f743a937baa16fa57dcefc348e228a"}
Nov 03 07:53:21.341921 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.341450    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:21.357439 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.357418    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:21.375206 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.375171    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z" podStartSLOduration=7.393274021 podStartE2EDuration="23.375160125s" totalImagesPullingTime="15.981886104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:21.375041094 +0000 UTC m=+23.734085796" watchObservedRunningTime="2026-11-03 07:53:21.375160125 +0000 UTC m=+23.734204828"
Nov 03 07:53:21.393354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.393326    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:21.544002 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.543981    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:21.746147 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.746044    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:21.885779 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:21.885748    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:22.139826 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.139752    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:22.139955 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.139765    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:22.139955 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.139868    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:22.139955 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:22.139867    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:22.140082 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:22.139956    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:22.140082 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:22.140030    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:22.151834 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.151811    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:22.343698 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.343669    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:22.344103 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.343773    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:22.354814 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.354785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:22.358536 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.358513    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:22.470958 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.470908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:22.944169 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:22.944050    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:23.186374 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:23.186348    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7zvx_16c1f0f2-b9b6-4659-97fb-9b3092e04f5f/dns-node-resolver/0.log"
Nov 03 07:53:23.537873 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:23.537843    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:23.650967 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:23.650941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:23.755956 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:23.755918    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:24.139864 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.139836    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:24.139864 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.139860    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:24.140116 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.139844    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:24.140116 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:24.139942    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:24.140116 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:24.140007    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:24.140116 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:24.140090    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:24.148104 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.148082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:24.186307 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.186286    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-srvsw_e18c61c8-db8c-49ca-8fbd-9378be7e9c51/node-ca/0.log"
Nov 03 07:53:24.338154 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.338118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:24.816654 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.816622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:24.940615 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:24.940594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:25.096910 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:25.096847    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:25.539814 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:25.539739    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:25.741134 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:25.741107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:26.139812 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:26.139783    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:26.140247 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:26.139843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:26.140247 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:26.139847    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:26.140247 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:26.139931    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:26.140247 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:26.140078    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:26.140247 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:26.140161    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:26.141698 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:26.141659    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:26.344428 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:26.344397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:26.897785 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:26.897719    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:27.011819 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:27.011769    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:27.127850 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:27.127825    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:27.346938 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:27.346906    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:27.744486 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:27.744457    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:28.139914 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:28.139816    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:28.139914 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:28.139817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:28.140419 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:28.140368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:28.140831 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:28.140815    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:28.140910 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:28.140894    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:28.140980 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:28.140958    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:28.141105 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:28.141052    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:28.543516 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:28.543487    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:28.740910 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:28.740874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:28.945953 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:28.945901    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:30.140402 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:30.140372    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:30.140916 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:30.140430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:30.140916 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:30.140434    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:30.140916 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:30.140521    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:30.140916 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:30.140613    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:30.140916 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:30.140708    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:31.749327 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:31.749161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:31.749724 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.749305    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:31.749724 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.749431    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret podName:342d6281-4bdd-4090-ac8b-717ca2942a0a nodeName:}" failed. No retries permitted until 2026-11-03 07:54:03.749416308 +0000 UTC m=+66.108460988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret") pod "global-pull-secret-syncer-898qp" (UID: "342d6281-4bdd-4090-ac8b-717ca2942a0a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:53:31.849917 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:31.849885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:31.850095 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.850032    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:31.850144 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.850114    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs podName:6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf nodeName:}" failed. No retries permitted until 2026-11-03 07:54:03.850099532 +0000 UTC m=+66.209144225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs") pod "network-metrics-daemon-m7nbf" (UID: "6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:53:31.950692 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:31.950654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:31.950853 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.950809    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:53:31.950853 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.950825    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:53:31.950853 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.950836    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5qq5w for pod openshift-network-diagnostics/network-check-target-k5mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:31.950950 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:31.950883    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w podName:76974834-71dc-4488-b393-ed2730c0276a nodeName:}" failed. No retries permitted until 2026-11-03 07:54:03.950870354 +0000 UTC m=+66.309915035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qq5w" (UniqueName: "kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w") pod "network-check-target-k5mr4" (UID: "76974834-71dc-4488-b393-ed2730c0276a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:53:32.143027 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:32.142959    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:32.143191 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:32.143054    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:32.143191 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:32.143080    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:32.143191 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:32.143083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:32.143191 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:32.143159    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:32.143364 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:32.143257    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:34.143094 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:34.143049    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:34.143495 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:34.143135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:34.143495 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:34.143160    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:34.143495 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:34.143223    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:34.143495 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:34.143263    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:34.143495 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:34.143337    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:35.576419 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:35.576388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:35.683096 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:35.683035    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:35.785275 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:35.785246    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:36.065861 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:36.065823    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:36.140273 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:36.140241    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:36.140424 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:36.140359    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:36.140424 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:36.140363    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:36.140512 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:36.140443    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:36.140512 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:36.140457    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:36.140583 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:36.140549    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:36.171119 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:36.171081    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:36.277273 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:36.277242    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:37.086759 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:37.086703    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:37.285656 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:37.285627    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:37.398905 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:37.398840    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:38.143333 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.143305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:38.143784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.143432    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:38.143784 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:38.143464    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:38.143784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.143468    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:38.143784 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:38.143531    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:38.143784 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:38.143610    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:38.313404 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.313370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:38.652604 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.652560    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5mr4"]
Nov 03 07:53:38.652815 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.652719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:38.653922 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:38.652839    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:38.653922 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.653423    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-898qp"]
Nov 03 07:53:38.653922 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.653525    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:38.653922 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:38.653618    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:38.656226 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.656201    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m7nbf"]
Nov 03 07:53:38.656328 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.656312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:38.656434 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:38.656412    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:38.983139 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:38.983119    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7zvx_16c1f0f2-b9b6-4659-97fb-9b3092e04f5f/dns-node-resolver/0.log"
Nov 03 07:53:40.140664 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:40.140632    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:40.141194 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:40.140632    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:40.141194 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:40.140746    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:40.141194 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:40.140747    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:40.141194 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:40.140807    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:40.141194 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:40.140916    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:40.183286 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:40.183260    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-srvsw_e18c61c8-db8c-49ca-8fbd-9378be7e9c51/node-ca/0.log"
Nov 03 07:53:40.196507 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:40.196483    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:40.304860 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:40.304826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:40.409562 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:40.409486    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:41.896818 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:41.896787    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:42.006216 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.006186    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:42.139970 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.139940    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:42.139970 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.139971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:42.140202 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.139982    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:42.140202 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:42.140101    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7nbf" podUID="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf"
Nov 03 07:53:42.140439 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:42.140417    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-898qp" podUID="342d6281-4bdd-4090-ac8b-717ca2942a0a"
Nov 03 07:53:42.143776 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:42.140735    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5mr4" podUID="76974834-71dc-4488-b393-ed2730c0276a"
Nov 03 07:53:42.652813 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.652783    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:42.847785 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.847713    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeReady"
Nov 03 07:53:42.847917 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.847822    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 07:53:42.890634 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.890596    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4p5vg"]
Nov 03 07:53:42.892511 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.892488    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:42.893470 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.893442    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-45pld"]
Nov 03 07:53:42.894823 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.894809    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:42.900225 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.900006    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 07:53:42.900533 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.900235    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 07:53:42.900533 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.900235    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 07:53:42.900533 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.900311    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 07:53:42.901854 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.901830    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ghck6\""
Nov 03 07:53:42.901955 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.901889    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 07:53:42.901955 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.901917    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjbhz\""
Nov 03 07:53:42.904415 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.904396    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4p5vg"]
Nov 03 07:53:42.909866 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.909843    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45pld"]
Nov 03 07:53:42.918077 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.918033    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lfvbs"]
Nov 03 07:53:42.919831 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.919817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:42.931824 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:42.931804    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfvbs"]
Nov 03 07:53:43.000830 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.000784    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 07:53:43.001018 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.000957    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 07:53:43.001018 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.000998    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 07:53:43.001018 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.001006    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5spf6\""
Nov 03 07:53:43.001171 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.001112    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 07:53:43.034397 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d680dcf8-8cd0-488b-b8b1-b66368817faf-cert\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:43.034397 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb54163a-2ad1-4b02-93d6-f8eac877da15-metrics-tls\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.034539 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de01be03-1a51-4ada-9b3c-498276ad5677-crio-socket\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.034539 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54163a-2ad1-4b02-93d6-f8eac877da15-config-volume\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.034609 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034539    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzwgw\" (UniqueName: \"kubernetes.io/projected/d680dcf8-8cd0-488b-b8b1-b66368817faf-kube-api-access-kzwgw\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:43.034609 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de01be03-1a51-4ada-9b3c-498276ad5677-data-volume\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.034680 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxjg\" (UniqueName: \"kubernetes.io/projected/bb54163a-2ad1-4b02-93d6-f8eac877da15-kube-api-access-dlxjg\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.034713 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de01be03-1a51-4ada-9b3c-498276ad5677-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.034713 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb54163a-2ad1-4b02-93d6-f8eac877da15-tmp-dir\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.034784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de01be03-1a51-4ada-9b3c-498276ad5677-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.034784 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.034746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzw74\" (UniqueName: \"kubernetes.io/projected/de01be03-1a51-4ada-9b3c-498276ad5677-kube-api-access-dzw74\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.135424 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135348    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwgw\" (UniqueName: \"kubernetes.io/projected/d680dcf8-8cd0-488b-b8b1-b66368817faf-kube-api-access-kzwgw\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:43.135424 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de01be03-1a51-4ada-9b3c-498276ad5677-data-volume\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxjg\" (UniqueName: \"kubernetes.io/projected/bb54163a-2ad1-4b02-93d6-f8eac877da15-kube-api-access-dlxjg\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de01be03-1a51-4ada-9b3c-498276ad5677-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb54163a-2ad1-4b02-93d6-f8eac877da15-tmp-dir\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de01be03-1a51-4ada-9b3c-498276ad5677-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzw74\" (UniqueName: \"kubernetes.io/projected/de01be03-1a51-4ada-9b3c-498276ad5677-kube-api-access-dzw74\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d680dcf8-8cd0-488b-b8b1-b66368817faf-cert\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb54163a-2ad1-4b02-93d6-f8eac877da15-metrics-tls\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.135631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de01be03-1a51-4ada-9b3c-498276ad5677-crio-socket\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.135998 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54163a-2ad1-4b02-93d6-f8eac877da15-config-volume\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.135998 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135903    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb54163a-2ad1-4b02-93d6-f8eac877da15-tmp-dir\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.135998 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.135904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de01be03-1a51-4ada-9b3c-498276ad5677-crio-socket\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.136568 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.136391    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de01be03-1a51-4ada-9b3c-498276ad5677-data-volume\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.136692 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.136394    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de01be03-1a51-4ada-9b3c-498276ad5677-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.136692 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.136432    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54163a-2ad1-4b02-93d6-f8eac877da15-config-volume\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.139464 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.139441    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb54163a-2ad1-4b02-93d6-f8eac877da15-metrics-tls\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.139571 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.139479    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de01be03-1a51-4ada-9b3c-498276ad5677-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.139571 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.139502    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d680dcf8-8cd0-488b-b8b1-b66368817faf-cert\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:43.144632 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.144591    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxjg\" (UniqueName: \"kubernetes.io/projected/bb54163a-2ad1-4b02-93d6-f8eac877da15-kube-api-access-dlxjg\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.144767 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.144747    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwgw\" (UniqueName: \"kubernetes.io/projected/d680dcf8-8cd0-488b-b8b1-b66368817faf-kube-api-access-kzwgw\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:43.145395 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.145369    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzw74\" (UniqueName: \"kubernetes.io/projected/de01be03-1a51-4ada-9b3c-498276ad5677-kube-api-access-dzw74\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.205853 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.205820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:43.210942 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.210915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:53:43.231486 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.231458    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:53:43.397110 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.397023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:43.513036 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.513008    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45pld"]
Nov 03 07:53:43.516207 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:43.516180    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd680dcf8_8cd0_488b_b8b1_b66368817faf.slice/crio-d8b340fa765bb9d414fb9355c0c7314ba243407539c8d5d6e309910061598c01 WatchSource:0}: Error finding container d8b340fa765bb9d414fb9355c0c7314ba243407539c8d5d6e309910061598c01: Status 404 returned error can't find the container with id d8b340fa765bb9d414fb9355c0c7314ba243407539c8d5d6e309910061598c01
Nov 03 07:53:43.519008 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.518983    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4p5vg"]
Nov 03 07:53:43.519623 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:43.519603    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfvbs"]
Nov 03 07:53:43.524471 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:43.524450    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde01be03_1a51_4ada_9b3c_498276ad5677.slice/crio-a6792a1a69a64bd7eeb652ac9685ae29213a12dec8805ff6b29342ac559512dc WatchSource:0}: Error finding container a6792a1a69a64bd7eeb652ac9685ae29213a12dec8805ff6b29342ac559512dc: Status 404 returned error can't find the container with id a6792a1a69a64bd7eeb652ac9685ae29213a12dec8805ff6b29342ac559512dc
Nov 03 07:53:43.525042 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:43.525025    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb54163a_2ad1_4b02_93d6_f8eac877da15.slice/crio-c6022d71402a00d2c235b662c1ef924357fc899b05012dda33d9480c6e4fb1a5 WatchSource:0}: Error finding container c6022d71402a00d2c235b662c1ef924357fc899b05012dda33d9480c6e4fb1a5: Status 404 returned error can't find the container with id c6022d71402a00d2c235b662c1ef924357fc899b05012dda33d9480c6e4fb1a5
Nov 03 07:53:44.139982 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.139946    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:53:44.140984 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.140220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:53:44.140984 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.140389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:53:44.145425 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.145390    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 07:53:44.145544 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.145451    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 07:53:44.147761 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.147740    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\""
Nov 03 07:53:44.147857 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.147812    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\""
Nov 03 07:53:44.147913 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.147901    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 07:53:44.147966 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.147955    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\""
Nov 03 07:53:44.148044 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.148032    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 07:53:44.389531 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.389490    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45pld" event={"ID":"d680dcf8-8cd0-488b-b8b1-b66368817faf","Type":"ContainerStarted","Data":"d8b340fa765bb9d414fb9355c0c7314ba243407539c8d5d6e309910061598c01"}
Nov 03 07:53:44.390617 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.390555    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4p5vg" event={"ID":"bb54163a-2ad1-4b02-93d6-f8eac877da15","Type":"ContainerStarted","Data":"c6022d71402a00d2c235b662c1ef924357fc899b05012dda33d9480c6e4fb1a5"}
Nov 03 07:53:44.392049 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.392030    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfvbs" event={"ID":"de01be03-1a51-4ada-9b3c-498276ad5677","Type":"ContainerStarted","Data":"1540b66af23bc8050bd4fab6572854654a637ebdc4125b44e88774df059c3c3a"}
Nov 03 07:53:44.392161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.392098    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfvbs" event={"ID":"de01be03-1a51-4ada-9b3c-498276ad5677","Type":"ContainerStarted","Data":"b61e76664e39c41cb151ff6bcccb3682e2797d60132edc9c33f343875aac4d56"}
Nov 03 07:53:44.392161 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.392130    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfvbs" event={"ID":"de01be03-1a51-4ada-9b3c-498276ad5677","Type":"ContainerStarted","Data":"a6792a1a69a64bd7eeb652ac9685ae29213a12dec8805ff6b29342ac559512dc"}
Nov 03 07:53:44.422763 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.422725    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:44.528517 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:44.528472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:45.368827 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.368789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:45.504974 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.504946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:45.546581 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.546534    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"]
Nov 03 07:53:45.549390 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.549371    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.558985 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.558962    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 07:53:45.561329 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.561285    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 07:53:45.561651 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.561621    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 07:53:45.562926 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.562875    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"]
Nov 03 07:53:45.563014 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.562988    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wvjx7\""
Nov 03 07:53:45.563100 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.563020    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 07:53:45.563982 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.563686    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 07:53:45.589748 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.589720    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vfxjv"]
Nov 03 07:53:45.592284 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.592262    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.615354 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.615311    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 07:53:45.615634 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.615611    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8t4tj\""
Nov 03 07:53:45.615634 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.615621    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 07:53:45.652442 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.652415    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:45.653289 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.653264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.653371 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.653289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.653371 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.653318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b98521-a087-4c0d-a301-a49080ff65b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.653371 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.653334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngrbz\" (UniqueName: \"kubernetes.io/projected/c8b98521-a087-4c0d-a301-a49080ff65b5-kube-api-access-ngrbz\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.659304 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.659282    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 07:53:45.754307 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-sys\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754444 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-wtmp\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754512 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-textfile\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754512 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-tls\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754512 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-root\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754658 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754658 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw946\" (UniqueName: \"kubernetes.io/projected/8387d126-b80b-43e1-978b-4ee863c4ddee-kube-api-access-lw946\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754756 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.754756 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.754756 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-accelerators-collector-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.754904 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b98521-a087-4c0d-a301-a49080ff65b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.754904 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngrbz\" (UniqueName: \"kubernetes.io/projected/c8b98521-a087-4c0d-a301-a49080ff65b5-kube-api-access-ngrbz\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.754904 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.754845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-metrics-client-ca\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.755042 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:45.754991    2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Nov 03 07:53:45.755130 ip-10-0-140-232 kubenswrapper[2574]: E1103 07:53:45.755078    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls podName:c8b98521-a087-4c0d-a301-a49080ff65b5 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:46.255031065 +0000 UTC m=+48.614075748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls") pod "openshift-state-metrics-7bd7489957-98nq2" (UID: "c8b98521-a087-4c0d-a301-a49080ff65b5") : secret "openshift-state-metrics-tls" not found
Nov 03 07:53:45.756016 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.755994    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b98521-a087-4c0d-a301-a49080ff65b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.758720 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.758699    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.769468 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.769442    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngrbz\" (UniqueName: \"kubernetes.io/projected/c8b98521-a087-4c0d-a301-a49080ff65b5-kube-api-access-ngrbz\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:45.771975 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.771953    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-wtmp\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-textfile\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-tls\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-root\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw946\" (UniqueName: \"kubernetes.io/projected/8387d126-b80b-43e1-978b-4ee863c4ddee-kube-api-access-lw946\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-accelerators-collector-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-metrics-client-ca\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-sys\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856382    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-sys\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.856579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.856499    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-wtmp\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.863265 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.857186    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-textfile\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.863265 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.860993    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-root\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.863265 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.861694    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-metrics-client-ca\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.863265 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.861724    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-accelerators-collector-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.865846 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.865823    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-tls\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.866987 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.866966    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.869211 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.869187    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw946\" (UniqueName: \"kubernetes.io/projected/8387d126-b80b-43e1-978b-4ee863c4ddee-kube-api-access-lw946\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.885886 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.885700    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:45.916623 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.916602    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:53:45.927448 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:45.927425    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8387d126_b80b_43e1_978b_4ee863c4ddee.slice/crio-cfe57c65270076254d8f4a998bb86c20193871b9e7dc1c741f66eccdb8c5c94f WatchSource:0}: Error finding container cfe57c65270076254d8f4a998bb86c20193871b9e7dc1c741f66eccdb8c5c94f: Status 404 returned error can't find the container with id cfe57c65270076254d8f4a998bb86c20193871b9e7dc1c741f66eccdb8c5c94f
Nov 03 07:53:45.990989 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:45.990952    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:46.198040 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.198006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:46.259850 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.259815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:46.262739 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.262710    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:46.399498 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.399462    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4p5vg" event={"ID":"bb54163a-2ad1-4b02-93d6-f8eac877da15","Type":"ContainerStarted","Data":"7a1f2572895f5fc769960b244f21d40bd497d93b0c1ad0ef9b0e867318ceee2e"}
Nov 03 07:53:46.399959 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.399521    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4p5vg" event={"ID":"bb54163a-2ad1-4b02-93d6-f8eac877da15","Type":"ContainerStarted","Data":"60b32df7df305392fa25d303367e859654a71ea65934702358bec402852665a6"}
Nov 03 07:53:46.399959 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.399589    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:46.400817 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.400789    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vfxjv" event={"ID":"8387d126-b80b-43e1-978b-4ee863c4ddee","Type":"ContainerStarted","Data":"cfe57c65270076254d8f4a998bb86c20193871b9e7dc1c741f66eccdb8c5c94f"}
Nov 03 07:53:46.418203 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.418156    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-4p5vg" podStartSLOduration=2.502420347 podStartE2EDuration="4.418141036s" totalImagesPullingTime="1.915720689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:46.417018477 +0000 UTC m=+48.776063181" watchObservedRunningTime="2026-11-03 07:53:46.418141036 +0000 UTC m=+48.777185769"
Nov 03 07:53:46.467399 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.467338    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:53:46.604363 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.604332    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:46.658631 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.658599    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:53:46.661608 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.661053    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.666197 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666170    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 07:53:46.666197 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666186    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 07:53:46.666351 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666250    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 07:53:46.666351 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666254    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 07:53:46.666351 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666289    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 07:53:46.666351 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666287    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 07:53:46.666351 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666292    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 07:53:46.666351 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666322    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 07:53:46.666532 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.666365    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 07:53:46.668316 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.668297    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c4vhz\""
Nov 03 07:53:46.677644 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.677618    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:53:46.721956 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.721903    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:46.762939 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.762910    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763086 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.762964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763086 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.762992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763188 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763188 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763276 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763230    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xn2\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763325 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763371 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763459 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.763579 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.763546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.769742 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:46.769703    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b98521_a087_4c0d_a301_a49080ff65b5.slice/crio-e838e5117fa1ca9f0e5d54ece4671b728057b3f2ec68515fa9e0090af5c1b0ce WatchSource:0}: Error finding container e838e5117fa1ca9f0e5d54ece4671b728057b3f2ec68515fa9e0090af5c1b0ce: Status 404 returned error can't find the container with id e838e5117fa1ca9f0e5d54ece4671b728057b3f2ec68515fa9e0090af5c1b0ce
Nov 03 07:53:46.771559 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.771535    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"]
Nov 03 07:53:46.864559 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864663 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864663 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xn2\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864663 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864663 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864663 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864656    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.864927 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.864805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.865278 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.865240    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.865941 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.865913    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.867530 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.867127    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.869777 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.869753    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.870771 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.870748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.872240 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.872181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.872334 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.872301    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.872721 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.872691    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.872848 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.872822    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.874052 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.873944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.874052 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.874018    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.874344 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.874321    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.877945 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.877879    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xn2\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:46.979890 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:46.979818    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:53:47.275421 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.275332    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:53:47.281658 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:47.281620    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84caeca2_05a1_4588_b70e_c85607c1d219.slice/crio-e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9 WatchSource:0}: Error finding container e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9: Status 404 returned error can't find the container with id e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9
Nov 03 07:53:47.406522 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.406490    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerStarted","Data":"e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9"}
Nov 03 07:53:47.408662 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.408636    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2" event={"ID":"c8b98521-a087-4c0d-a301-a49080ff65b5","Type":"ContainerStarted","Data":"54d947b60f5b5dbfa254810b48ca2d04eb21acefe743156bb860ee059d8dff4c"}
Nov 03 07:53:47.408794 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.408687    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2" event={"ID":"c8b98521-a087-4c0d-a301-a49080ff65b5","Type":"ContainerStarted","Data":"8cb1310bacf1f013e8bc2d76a93759957de28fa69a5e78d89241d44e8f59cdfa"}
Nov 03 07:53:47.408794 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.408711    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2" event={"ID":"c8b98521-a087-4c0d-a301-a49080ff65b5","Type":"ContainerStarted","Data":"e838e5117fa1ca9f0e5d54ece4671b728057b3f2ec68515fa9e0090af5c1b0ce"}
Nov 03 07:53:47.410487 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.410462    2574 generic.go:356] "Generic (PLEG): container finished" podID="8387d126-b80b-43e1-978b-4ee863c4ddee" containerID="d3f20f54104583c7677d08e014fa8f488539570d92544fb5c1a275627be3cace" exitCode=0
Nov 03 07:53:47.410576 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.410553    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vfxjv" event={"ID":"8387d126-b80b-43e1-978b-4ee863c4ddee","Type":"ContainerDied","Data":"d3f20f54104583c7677d08e014fa8f488539570d92544fb5c1a275627be3cace"}
Nov 03 07:53:47.412978 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.412956    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfvbs" event={"ID":"de01be03-1a51-4ada-9b3c-498276ad5677","Type":"ContainerStarted","Data":"17fea45ff95738d203d9131251952027af8b4021ad235a5c7d89e04e859c7813"}
Nov 03 07:53:47.447756 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:47.447706    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lfvbs" podStartSLOduration=2.624141828 podStartE2EDuration="5.447689757s" totalImagesPullingTime="2.823547929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:47.446078395 +0000 UTC m=+49.805123097" watchObservedRunningTime="2026-11-03 07:53:47.447689757 +0000 UTC m=+49.806734457"
Nov 03 07:53:48.417690 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.417653    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45pld" event={"ID":"d680dcf8-8cd0-488b-b8b1-b66368817faf","Type":"ContainerStarted","Data":"47306d2902a4ad4cf77cb44a296ea15f02737339ebe1c662c06c277e9b0f8933"}
Nov 03 07:53:48.424006 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.423980    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vfxjv" event={"ID":"8387d126-b80b-43e1-978b-4ee863c4ddee","Type":"ContainerStarted","Data":"90a513621cefc1a55b5329ecb4c708d16567e01757ecbff4f278e306b3db9827"}
Nov 03 07:53:48.424172 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.424086    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vfxjv" event={"ID":"8387d126-b80b-43e1-978b-4ee863c4ddee","Type":"ContainerStarted","Data":"5bf1e183dd87258b4300f24c56161726f36c6062835e4cbe15d76b7c4a3ee08a"}
Nov 03 07:53:48.426194 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.426170    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2" event={"ID":"c8b98521-a087-4c0d-a301-a49080ff65b5","Type":"ContainerStarted","Data":"3a994c25f0e29d6e2285a6497524a091637c8a68be2ab4ff153ae00b7b335014"}
Nov 03 07:53:48.435594 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.435546    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-45pld" podStartSLOduration=1.927851548 podStartE2EDuration="6.435535135s" totalImagesPullingTime="4.507683587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:48.43466288 +0000 UTC m=+50.793707581" watchObservedRunningTime="2026-11-03 07:53:48.435535135 +0000 UTC m=+50.794579836"
Nov 03 07:53:48.458710 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.458661    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vfxjv" podStartSLOduration=2.187232281 podStartE2EDuration="3.458646038s" totalImagesPullingTime="1.188864434s" totalInitContainerRuntime="82.549323ms" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:48.457621019 +0000 UTC m=+50.816665724" watchObservedRunningTime="2026-11-03 07:53:48.458646038 +0000 UTC m=+50.817690734"
Nov 03 07:53:48.476951 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.476900    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2" podStartSLOduration=2.114758974 podStartE2EDuration="3.476885987s" totalImagesPullingTime="1.362127013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:48.475814676 +0000 UTC m=+50.834859379" watchObservedRunningTime="2026-11-03 07:53:48.476885987 +0000 UTC m=+50.835930689"
Nov 03 07:53:48.576565 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.576539    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:48.687617 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.687580    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:48.898516 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:48.898442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:49.430208 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.430180    2574 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700" exitCode=0
Nov 03 07:53:49.430568 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.430287    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700"}
Nov 03 07:53:49.681703 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.681666    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:49.786834 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.786751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:49.983974 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.983911    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-688749c497-5dxnm"]
Nov 03 07:53:49.986924 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.986907    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:49.991369 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.991343    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 07:53:49.991482 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.991389    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 07:53:49.996134 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:49.996113    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-688749c497-5dxnm"]
Nov 03 07:53:50.025861 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.025837    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 07:53:50.025968 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.025874    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 07:53:50.025968 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.025874    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pdlxh\""
Nov 03 07:53:50.025968 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.025928    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d835iev2t6ar\""
Nov 03 07:53:50.091533 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.091501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-tls\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.091667 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.091542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.091667 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.091615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9600bd0a-077f-40c5-b09d-eac5da61049b-audit-log\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.091667 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.091653    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-client-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.091814 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.091678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-client-certs\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.091814 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.091702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-metrics-server-audit-profiles\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.091814 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.091748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77f5l\" (UniqueName: \"kubernetes.io/projected/9600bd0a-077f-40c5-b09d-eac5da61049b-kube-api-access-77f5l\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193034 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.192996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77f5l\" (UniqueName: \"kubernetes.io/projected/9600bd0a-077f-40c5-b09d-eac5da61049b-kube-api-access-77f5l\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193200 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193108    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-tls\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193200 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193313 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9600bd0a-077f-40c5-b09d-eac5da61049b-audit-log\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193386 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-client-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193448 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-client-certs\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193448 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-metrics-server-audit-profiles\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.193696 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193669    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9600bd0a-077f-40c5-b09d-eac5da61049b-audit-log\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.194015 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.193913    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.194233 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.194167    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-metrics-server-audit-profiles\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.195671 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.195644    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-tls\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.195764 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.195715    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-client-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.195850 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.195829    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-client-certs\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.201734 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.201696    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77f5l\" (UniqueName: \"kubernetes.io/projected/9600bd0a-077f-40c5-b09d-eac5da61049b-kube-api-access-77f5l\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.298159 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.298085    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:53:50.391785 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.391749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:50.501786 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.501748    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:50.577947 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.577871    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-688749c497-5dxnm"]
Nov 03 07:53:50.580779 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:50.580754    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9600bd0a_077f_40c5_b09d_eac5da61049b.slice/crio-80b785777e04974bee3960968c36628f7ff27d00b92e31cbebf0503e391c371a WatchSource:0}: Error finding container 80b785777e04974bee3960968c36628f7ff27d00b92e31cbebf0503e391c371a: Status 404 returned error can't find the container with id 80b785777e04974bee3960968c36628f7ff27d00b92e31cbebf0503e391c371a
Nov 03 07:53:50.615033 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.615003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:50.810016 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.809988    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"]
Nov 03 07:53:50.814590 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.814566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.821239 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.821218    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 07:53:50.825124 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.825099    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 07:53:50.825230 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.825172    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 07:53:50.825301 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.825273    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 07:53:50.826904 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.826885    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 07:53:50.827590 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.827548    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4jdjw\""
Nov 03 07:53:50.829044 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.828995    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"]
Nov 03 07:53:50.899358 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-metrics-client-ca\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.899511 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.899511 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.899639 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899515    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.899639 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-federate-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.899639 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-serving-certs-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.899786 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.899786 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.899743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hb6\" (UniqueName: \"kubernetes.io/projected/37e9ee43-9770-42c8-9064-92437099e68a-kube-api-access-w6hb6\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:50.927649 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:50.927620    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 07:53:51.000142 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-metrics-client-ca\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000142 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000146    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000359 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000359 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000359 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000330    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-federate-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000593 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-serving-certs-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000593 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000417    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000593 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hb6\" (UniqueName: \"kubernetes.io/projected/37e9ee43-9770-42c8-9064-92437099e68a-kube-api-access-w6hb6\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.000899 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.000876    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-metrics-client-ca\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.001483 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.001431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.001689 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.001661    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-serving-certs-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.002869 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.002849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.002950 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.002910    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.002998 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.002947    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.003124 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.003108    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-federate-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.008967 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.008949    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hb6\" (UniqueName: \"kubernetes.io/projected/37e9ee43-9770-42c8-9064-92437099e68a-kube-api-access-w6hb6\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.134799 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.134732    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:53:51.414852 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.414766    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"]
Nov 03 07:53:51.420085 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:53:51.420037    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e9ee43_9770_42c8_9064_92437099e68a.slice/crio-5461af58acccb5b14fc3121dd8aae6be55ba422f1a9f1a14b7887cb42ef67497 WatchSource:0}: Error finding container 5461af58acccb5b14fc3121dd8aae6be55ba422f1a9f1a14b7887cb42ef67497: Status 404 returned error can't find the container with id 5461af58acccb5b14fc3121dd8aae6be55ba422f1a9f1a14b7887cb42ef67497
Nov 03 07:53:51.439751 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.439727    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l" event={"ID":"37e9ee43-9770-42c8-9064-92437099e68a","Type":"ContainerStarted","Data":"5461af58acccb5b14fc3121dd8aae6be55ba422f1a9f1a14b7887cb42ef67497"}
Nov 03 07:53:51.440994 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.440970    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-688749c497-5dxnm" event={"ID":"9600bd0a-077f-40c5-b09d-eac5da61049b","Type":"ContainerStarted","Data":"80b785777e04974bee3960968c36628f7ff27d00b92e31cbebf0503e391c371a"}
Nov 03 07:53:51.895432 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:51.895370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:52.448968 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.448932    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerStarted","Data":"6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb"}
Nov 03 07:53:52.449219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.449033    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerStarted","Data":"76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123"}
Nov 03 07:53:52.449219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.449096    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerStarted","Data":"bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a"}
Nov 03 07:53:52.449219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.449142    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerStarted","Data":"e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0"}
Nov 03 07:53:52.449219 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.449178    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerStarted","Data":"9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea"}
Nov 03 07:53:52.450825 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.450799    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-688749c497-5dxnm" event={"ID":"9600bd0a-077f-40c5-b09d-eac5da61049b","Type":"ContainerStarted","Data":"b3c76f0a00efabc4581e39b9236a1241e4235a25db14d1721f919e08da4205ea"}
Nov 03 07:53:52.641780 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.641670    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:52.749858 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.749814    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:52.904515 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:52.904438    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:53.021317 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:53.021287    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:53.461494 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:53.461456    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerStarted","Data":"c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0"}
Nov 03 07:53:53.504549 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:53.504500    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.19680362 podStartE2EDuration="7.504480637s" totalImagesPullingTime="4.295463168s" totalInitContainerRuntime="12.213849ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:53.501710419 +0000 UTC m=+55.860755122" watchObservedRunningTime="2026-11-03 07:53:53.504480637 +0000 UTC m=+55.863525343"
Nov 03 07:53:53.504932 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:53.504891    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-688749c497-5dxnm" podStartSLOduration=2.792798087 podStartE2EDuration="4.504880151s" totalImagesPullingTime="1.712082064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:52.474008959 +0000 UTC m=+54.833053662" watchObservedRunningTime="2026-11-03 07:53:53.504880151 +0000 UTC m=+55.863924853"
Nov 03 07:53:54.358491 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:54.358467    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:53:54.466707 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:54.466672    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l" event={"ID":"37e9ee43-9770-42c8-9064-92437099e68a","Type":"ContainerStarted","Data":"b5cea8d68bec073373a4845ddc06af71339a472ad83821553180318a9c31246a"}
Nov 03 07:53:54.466849 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:54.466739    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l" event={"ID":"37e9ee43-9770-42c8-9064-92437099e68a","Type":"ContainerStarted","Data":"7c78ef5be6b9ef7c945eab38c0ec4abe0a5da01821b1ed22d5f85493ef722149"}
Nov 03 07:53:54.466849 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:54.466759    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l" event={"ID":"37e9ee43-9770-42c8-9064-92437099e68a","Type":"ContainerStarted","Data":"6cd973545a590d1fe7ad9976124168f12af558ac9fcf7ba064094fce58a1a561"}
Nov 03 07:53:54.492539 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:54.492499    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l" podStartSLOduration=2.181712015 podStartE2EDuration="4.492485168s" totalImagesPullingTime="2.310773153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:54.489846696 +0000 UTC m=+56.848891399" watchObservedRunningTime="2026-11-03 07:53:54.492485168 +0000 UTC m=+56.851529881"
Nov 03 07:53:55.526483 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:55.526444    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:55.640581 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:55.640552    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:56.414985 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:56.414960    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:53:56.891597 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:56.891555    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:57.004471 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:57.004441    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:57.707322 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:57.707289    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:57.821514 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:57.821483    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:58.834657 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:58.834626    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:58.948104 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:58.948050    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:59.388461 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:59.388402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:59.513890 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:59.513853    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:59.625875 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:59.625819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:53:59.735667 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:53:59.735626    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:00.376779 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:00.376747    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:00.494290 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:00.494263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:03.820041 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.819985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:03.823863 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.823844    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 07:54:03.832756 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.832734    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:03.921235 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.921188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:54:03.925078 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.925041    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 07:54:03.934461 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.934441    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:54:03.969043 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.969017    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\""
Nov 03 07:54:03.975971 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.975948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:03.976819 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.976796    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\""
Nov 03 07:54:03.983346 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:03.983327    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:54:04.022603 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.022467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:04.026132 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.026103    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 07:54:04.036840 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.036817    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 07:54:04.046780 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.046757    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:04.249260 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.249231    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-898qp"]
Nov 03 07:54:04.252011 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:54:04.251985    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342d6281_4bdd_4090_ac8b_717ca2942a0a.slice/crio-4890a77387a12b66496d05e2b6945131e9bb41c66eb499151b9fd7d7cad91380 WatchSource:0}: Error finding container 4890a77387a12b66496d05e2b6945131e9bb41c66eb499151b9fd7d7cad91380: Status 404 returned error can't find the container with id 4890a77387a12b66496d05e2b6945131e9bb41c66eb499151b9fd7d7cad91380
Nov 03 07:54:04.255834 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.255813    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m7nbf"]
Nov 03 07:54:04.258353 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:54:04.258331    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5d72b7_9d7e_42b2_9a2e_8c532c7522cf.slice/crio-0a713597c05f9b478cf4d24047b99039ce83e00a81fd49d6d83dd52ec839128a WatchSource:0}: Error finding container 0a713597c05f9b478cf4d24047b99039ce83e00a81fd49d6d83dd52ec839128a: Status 404 returned error can't find the container with id 0a713597c05f9b478cf4d24047b99039ce83e00a81fd49d6d83dd52ec839128a
Nov 03 07:54:04.261338 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.261321    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\""
Nov 03 07:54:04.268672 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.268657    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:04.496658 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.496627    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7nbf" event={"ID":"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf","Type":"ContainerStarted","Data":"0a713597c05f9b478cf4d24047b99039ce83e00a81fd49d6d83dd52ec839128a"}
Nov 03 07:54:04.497713 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.497694    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-898qp" event={"ID":"342d6281-4bdd-4090-ac8b-717ca2942a0a","Type":"ContainerStarted","Data":"4890a77387a12b66496d05e2b6945131e9bb41c66eb499151b9fd7d7cad91380"}
Nov 03 07:54:04.536851 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:04.536796    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5mr4"]
Nov 03 07:54:04.539544 ip-10-0-140-232 kubenswrapper[2574]: W1103 07:54:04.539523    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76974834_71dc_4488_b393_ed2730c0276a.slice/crio-c8da82b2c30c4cfd7f46f42b5d0b00779ba426d13aa7a797dbf2c2194b11dd0c WatchSource:0}: Error finding container c8da82b2c30c4cfd7f46f42b5d0b00779ba426d13aa7a797dbf2c2194b11dd0c: Status 404 returned error can't find the container with id c8da82b2c30c4cfd7f46f42b5d0b00779ba426d13aa7a797dbf2c2194b11dd0c
Nov 03 07:54:05.502995 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:05.502960    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5mr4" event={"ID":"76974834-71dc-4488-b393-ed2730c0276a","Type":"ContainerStarted","Data":"c8da82b2c30c4cfd7f46f42b5d0b00779ba426d13aa7a797dbf2c2194b11dd0c"}
Nov 03 07:54:05.569887 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:05.569814    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:05.686408 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:05.686374    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:06.507595 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:06.507553    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7nbf" event={"ID":"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf","Type":"ContainerStarted","Data":"e7245c61acb9d8f15d7e81cb21b3240242f688d9d93c2c471f47c488e710a0c0"}
Nov 03 07:54:06.507945 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:06.507617    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7nbf" event={"ID":"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf","Type":"ContainerStarted","Data":"a772677af761b45006fea09192b30318c93c0a89583826bbbbbb7639ca5f225d"}
Nov 03 07:54:06.524680 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:06.524630    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m7nbf" podStartSLOduration=66.784851528 podStartE2EDuration="1m8.524615886s" totalImagesPullingTime="1.739764358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:06.523835126 +0000 UTC m=+68.882879828" watchObservedRunningTime="2026-11-03 07:54:06.524615886 +0000 UTC m=+68.883660585"
Nov 03 07:54:06.650175 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:06.650140    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:06.767775 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:06.767697    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:09.195427 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:09.195393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:09.301206 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:09.301168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:09.520996 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:09.520910    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5mr4" event={"ID":"76974834-71dc-4488-b393-ed2730c0276a","Type":"ContainerStarted","Data":"9c9705dc2070555b8b491b0080af4173d44f24b689da130b6abdb3b504fc0fce"}
Nov 03 07:54:09.521170 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:09.521054    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:09.538220 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:09.538171    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k5mr4" podStartSLOduration=67.305370075 podStartE2EDuration="1m11.538153958s" totalImagesPullingTime="4.232783883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:09.536815758 +0000 UTC m=+71.895860464" watchObservedRunningTime="2026-11-03 07:54:09.538153958 +0000 UTC m=+71.897198661"
Nov 03 07:54:10.298826 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:10.298791    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:10.299175 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:10.298846    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:10.441137 ip-10-0-140-232 kubenswrapper[2574]: I1103 07:54:10.440978    2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:54:10.441161 ip-10-0-140-232 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 07:54:10.452485 ip-10-0-140-232 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 07:54:10.452651 ip-10-0-140-232 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 07:54:10.452802 ip-10-0-140-232 systemd[1]: kubelet.service: Consumed 4.094s CPU time.
Nov 03 07:54:10.465842 ip-10-0-140-232 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 07:54:10.525973 ip-10-0-140-232 kubenswrapper[6430]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:54:10.525973 ip-10-0-140-232 kubenswrapper[6430]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 07:54:10.525973 ip-10-0-140-232 kubenswrapper[6430]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:54:10.525973 ip-10-0-140-232 kubenswrapper[6430]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:54:10.528180 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528094    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.528180 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528179    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.528180 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528183    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528186    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528189    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528192    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528194    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528197    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528200    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528202    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528205    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528208    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528211    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528219    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528222    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528225    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528227    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528230    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528233    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528235    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528238    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.528278 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528241    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528244    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528247    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528249    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528253    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528256    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528259    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528261    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528264    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528267    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528270    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528273    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528275    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528278    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528281    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528284    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528286    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528289    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528292    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.528755 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528295    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528297    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528300    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528302    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528305    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528308    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528311    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528313    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528316    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528319    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528321    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528324    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528327    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528329    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528332    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528335    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528337    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528340    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528343    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528346    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.529248 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528349    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528351    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528354    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528356    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528359    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528362    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528364    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528367    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528370    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528372    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528375    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528377    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528380    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528382    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528385    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528387    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528390    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528393    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528395    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528398    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.529739 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528400    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528402    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528407    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528412    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528415    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528418    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528421    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528424    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528433    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528436    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528439    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528442    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528445    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528448    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528451    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528454    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528456    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528459    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528464    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.530265 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528467    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528470    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528473    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528475    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528478    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528483    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.528487    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529263    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529271    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529274    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529277    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529280    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529282    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529286    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529291    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529294    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529297    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529299    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529303    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.530731 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529305    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529309    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529311    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529314    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529317    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529320    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529322    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529325    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529327    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529330    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529332    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529335    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529337    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529341    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529343    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529346    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529348    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529351    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529354    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.531249 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529357    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529359    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529362    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529364    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529367    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529369    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529373    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529376    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529379    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529381    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529384    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529386    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529389    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529392    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529394    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529397    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529400    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529403    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529406    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.531722 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529409    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529411    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529414    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529416    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529419    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529422    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529425    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529429    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529432    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529435    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529438    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529441    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529443    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529446    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529448    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529451    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529453    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529456    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529459    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.532208 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529461    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529464    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529467    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529469    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529472    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529475    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529478    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529480    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529483    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529485    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529488    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529491    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529494    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529496    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529499    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529501    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529504    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529506    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529509    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529511    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529513    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.532711 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529516    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529519    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529522    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529524    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529530    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529534    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529536    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529539    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529542    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529544    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529546    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529549    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529552    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529554    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529557    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.529559    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530096    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530103    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530106    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.533224 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530110    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530115    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530118    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530122    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530126    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530129    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530131    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530134    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530137    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530139    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530142    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530145    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530147    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530150    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530153    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530156    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530159    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530165    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530167    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.533690 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530170    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530173    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530175    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530178    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530181    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530184    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530187    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530189    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530192    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530194    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530197    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530200    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530202    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530205    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530208    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530211    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530214    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530216    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530219    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530221    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.534175 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530224    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530227    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530229    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530232    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530234    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530237    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530239    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530243    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530247    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530251    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530255    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530258    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530261    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530263    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530266    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530269    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530272    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530274    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530277    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.534871 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530280    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530282    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530285    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530288    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530290    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530294    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530298    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530301    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530304    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530306    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530309    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530311    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530314    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530317    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530319    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530322    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530324    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530327    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530329    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530332    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.535664 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530334    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530336    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530340    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530344    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530346    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530349    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530352    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530354    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530357    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530359    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530362    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530364    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530367    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530369    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530372    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530374    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530377    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530379    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530382    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530385    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.536330 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530388    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530390    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530393    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530395    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.530399    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530490    6430 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530498    6430 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530504    6430 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530509    6430 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530513    6430 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530516    6430 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530521    6430 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530525    6430 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530528    6430 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530531    6430 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530537    6430 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530540    6430 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530543    6430 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530547    6430 flags.go:64] FLAG: --cgroup-root=""
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530550    6430 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530553    6430 flags.go:64] FLAG: --client-ca-file=""
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530556    6430 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530559    6430 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530562    6430 flags.go:64] FLAG: --cluster-domain=""
Nov 03 07:54:10.536822 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530565    6430 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530568    6430 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530572    6430 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530575    6430 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530579    6430 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530582    6430 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530585    6430 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530589    6430 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530592    6430 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530595    6430 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530598    6430 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530601    6430 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530604    6430 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530608    6430 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530611    6430 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530614    6430 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530617    6430 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530620    6430 flags.go:64] FLAG: --enable-server="true"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530623    6430 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530627    6430 flags.go:64] FLAG: --event-burst="100"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530630    6430 flags.go:64] FLAG: --event-qps="50"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530633    6430 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530636    6430 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530641    6430 flags.go:64] FLAG: --eviction-hard=""
Nov 03 07:54:10.537413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530650    6430 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530654    6430 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530657    6430 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530662    6430 flags.go:64] FLAG: --eviction-soft=""
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530665    6430 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530668    6430 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530671    6430 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530674    6430 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530677    6430 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530680    6430 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530683    6430 flags.go:64] FLAG: --feature-gates=""
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530692    6430 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530696    6430 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530699    6430 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530702    6430 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530705    6430 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530709    6430 flags.go:64] FLAG: --help="false"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530712    6430 flags.go:64] FLAG: --hostname-override="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530715    6430 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530718    6430 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530721    6430 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530724    6430 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530728    6430 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 07:54:10.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530731    6430 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530735    6430 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530738    6430 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530742    6430 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530745    6430 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530748    6430 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530751    6430 flags.go:64] FLAG: --kube-reserved=""
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530754    6430 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530758    6430 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530761    6430 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530764    6430 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530768    6430 flags.go:64] FLAG: --lock-file=""
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530772    6430 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530775    6430 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530778    6430 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530783    6430 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530786    6430 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530789    6430 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530793    6430 flags.go:64] FLAG: --logging-format="text"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530796    6430 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530799    6430 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530802    6430 flags.go:64] FLAG: --manifest-url=""
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530805    6430 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530809    6430 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530813    6430 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 07:54:10.538621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530817    6430 flags.go:64] FLAG: --max-pods="110"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530820    6430 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530823    6430 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530826    6430 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530829    6430 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530832    6430 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530835    6430 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530838    6430 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530845    6430 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530848    6430 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530851    6430 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530854    6430 flags.go:64] FLAG: --pod-cidr=""
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530857    6430 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530860    6430 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530863    6430 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530868    6430 flags.go:64] FLAG: --port="10250"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530871    6430 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530874    6430 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b7e0d9248434f705"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530877    6430 flags.go:64] FLAG: --qos-reserved=""
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530882    6430 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530885    6430 flags.go:64] FLAG: --register-node="true"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530888    6430 flags.go:64] FLAG: --register-with-taints=""
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530891    6430 flags.go:64] FLAG: --registry-burst="10"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530894    6430 flags.go:64] FLAG: --registry-qps="5"
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530897    6430 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530900    6430 flags.go:64] FLAG: --reserved-memory=""
Nov 03 07:54:10.539255 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530904    6430 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530907    6430 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530911    6430 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530913    6430 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530916    6430 flags.go:64] FLAG: --runonce="false"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530919    6430 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530923    6430 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530925    6430 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530928    6430 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530931    6430 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530934    6430 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530938    6430 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530941    6430 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530944    6430 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530947    6430 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530950    6430 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530952    6430 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530956    6430 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530959    6430 flags.go:64] FLAG: --system-cgroups=""
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530961    6430 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530967    6430 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530971    6430 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530974    6430 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530977    6430 flags.go:64] FLAG: --tls-min-version=""
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530980    6430 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 07:54:10.539933 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530985    6430 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530988    6430 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530991    6430 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530994    6430 flags.go:64] FLAG: --v="2"
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.530999    6430 flags.go:64] FLAG: --version="false"
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.531003    6430 flags.go:64] FLAG: --vmodule=""
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.531007    6430 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.531010    6430 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531165    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531169    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531172    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531175    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531179    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531182    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531184    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531187    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531190    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531192    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531195    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531198    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531200    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.540586 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531204    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531208    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531211    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531214    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531217    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531220    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531222    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531227    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531230    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531232    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531235    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531239    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531243    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531246    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531248    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531251    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531254    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531257    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531259    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531262    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.541123 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531265    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531268    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531270    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531273    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531275    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531278    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531281    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531284    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531286    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531289    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531291    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531294    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531296    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531299    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531301    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531304    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531306    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531309    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531312    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531315    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.541709 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531319    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531322    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531325    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531328    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531331    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531336    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531339    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531342    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531345    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531347    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531350    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531352    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531355    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531357    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531360    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531362    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531365    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531368    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531370    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.542546 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531373    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531375    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531377    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531380    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531383    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531385    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531388    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531390    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531393    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531395    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531398    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531400    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531403    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531405    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531408    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531410    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531413    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531416    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531418    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.543250 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531422    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531424    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531427    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531430    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531432    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531435    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531437    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531440    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531442    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531445    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531447    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531450    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531453    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531455    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531458    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.531484    6430 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:54:10.543809 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531518    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531523    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531526    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531529    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531532    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531535    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531537    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531540    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531542    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531545    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531547    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531550    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531552    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531555    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531557    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531560    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531562    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531565    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531568    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531571    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.544229 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531573    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531576    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531578    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531581    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531583    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531586    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531590    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531594    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531606    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531610    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531612    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531615    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531617    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531620    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531623    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531626    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531628    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531631    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531633    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.544736 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531636    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531638    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531641    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531643    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531646    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531648    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531651    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531653    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531656    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531658    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531660    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531664    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531666    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531669    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531672    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531674    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531677    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531680    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531682    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.545351 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531686    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531689    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531692    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531695    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531697    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531700    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531702    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531705    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531709    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531713    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531715    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531718    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531721    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531723    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531726    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531729    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531731    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531734    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531737    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.545912 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531740    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531743    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531745    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531748    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531750    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531753    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531755    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531758    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531761    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531763    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531766    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531768    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531771    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531775    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531777    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531781    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531785    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531788    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531791    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531793    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.546504 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531796    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531800    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531802    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531805    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531807    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531810    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531812    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531815    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.531819    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.537232    6430 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.537251    6430 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.537276    6430 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.537281    6430 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537344    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537349    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537353    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537357    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537360    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537363    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537366    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.546980 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537369    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537372    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537375    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537378    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537380    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537383    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537386    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537388    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537391    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537393    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537396    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537398    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537401    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537403    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537406    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537408    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537411    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537414    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537417    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.547489 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537420    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537423    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537426    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537429    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537432    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537435    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537437    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537440    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537442    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537445    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537447    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537450    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537452    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537455    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537457    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537460    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537462    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537465    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537468    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537471    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.547954 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537474    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537476    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537479    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537482    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537484    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537487    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537489    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537492    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537495    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537498    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537501    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537503    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537506    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537509    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537513    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537516    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537519    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537522    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537524    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.548452 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537528    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537533    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537536    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537539    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537542    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537544    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537547    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537550    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537553    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537555    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537558    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537560    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537563    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537566    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537569    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537572    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537575    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537577    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537580    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537582    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.548907 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537585    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537588    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537591    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537594    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537596    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537601    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537605    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537608    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537611    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537614    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537617    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537619    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537622    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537624    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537627    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537629    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537632    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537635    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537637    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.549413 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537640    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537643    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.537672    6430 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537821    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537825    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537828    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537832    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537836    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537840    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537843    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537847    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537850    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537852    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537855    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537858    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.549895 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537861    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537864    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537866    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537869    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537872    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537874    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537877    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537879    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537882    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537884    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537887    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537890    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537892    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537894    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537897    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537900    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537902    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537905    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537907    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.550553 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537910    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537912    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537915    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537917    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537920    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537922    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537925    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537928    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537930    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537933    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537936    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537938    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537941    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537944    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537946    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537949    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537952    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537954    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537957    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537960    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.551322 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537962    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537965    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537967    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537970    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537972    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537975    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537977    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537980    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537982    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537985    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537987    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537990    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537993    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537995    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.537997    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538000    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538003    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538005    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538008    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.552032 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538010    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538013    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538016    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538018    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538021    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538024    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538028    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538031    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538034    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538037    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538040    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538043    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538045    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538048    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538051    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538053    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538096    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538101    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538104    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538106    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.552559 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538109    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538111    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538114    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538116    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538119    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538122    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538124    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538127    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538129    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538132    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538134    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538137    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538139    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538142    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538145    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538148    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.553042 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.538173    6430 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538207    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538212    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538216    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538219    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538221    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538225    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538228    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538231    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538233    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538236    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538238    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538241    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538243    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538246    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538248    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538251    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538253    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538256    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:54:10.553445 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538258    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538261    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538263    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538267    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538271    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538274    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538276    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538279    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538282    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538284    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538287    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538289    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538294    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538296    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538299    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538301    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538304    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538309    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538312    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538314    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:54:10.553972 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538317    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538320    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538322    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538325    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538328    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538330    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538333    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538336    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538338    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538341    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538344    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538346    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538349    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538352    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538354    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538357    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538359    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538364    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538367    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:54:10.554521 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538369    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538372    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538374    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538377    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538380    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538382    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538386    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538388    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538391    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538394    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538398    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538401    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538404    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538406    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538409    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538412    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538414    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538417    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538419    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:54:10.555046 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538422    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538424    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538427    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538429    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538432    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538435    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538437    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538440    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538442    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538445    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538447    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538450    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538452    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538455    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538457    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538460    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538462    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538465    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538468    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538470    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:54:10.555533 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538473    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538476    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538480    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538484    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538486    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538489    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538491    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538494    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538497    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:10.538499    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.538565    6430 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.539569    6430 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.539645    6430 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.539822    6430 server.go:1014] "Starting client certificate rotation"
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.539968    6430 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.540017    6430 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 07:47:57 +0000 UTC" deadline="2028-08-12 13:06:02.069866812 +0000 UTC"
Nov 03 07:54:10.556020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.540054    6430 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15557h11m51.529814975s"
Nov 03 07:54:10.556530 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.540322    6430 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:54:10.556530 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.541441    6430 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:54:10.556530 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.543001    6430 log.go:25] "Validated CRI v1 runtime API"
Nov 03 07:54:10.556530 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.545079    6430 log.go:25] "Validated CRI v1 image API"
Nov 03 07:54:10.556530 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.545800    6430 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 07:54:10.556530 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.547951    6430 fs.go:135] Filesystem UUIDs: map[1ae67e76-f702-45f9-9d69-3355f221c940:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eb314d51-fa80-4249-aa78-95ede7f6766a:/dev/nvme0n1p3]
Nov 03 07:54:10.556710 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.547972    6430 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/079f77c5974971199aa139b9a43960bf5a69d77e69deebb7c4d26f5aab34a998/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/079f77c5974971199aa139b9a43960bf5a69d77e69deebb7c4d26f5aab34a998/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a713597c05f9b478cf4d24047b99039ce83e00a81fd49d6d83dd52ec839128a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a713597c05f9b478cf4d24047b99039ce83e00a81fd49d6d83dd52ec839128a/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/114512a799f2d7eb864d83a4aff0eefb65c2e5c3961aa714de039cc981fc883f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/114512a799f2d7eb864d83a4aff0eefb65c2e5c3961aa714de039cc981fc883f/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23de0469b8e597085dccb0d1c2b1d399002ff65d13c757f69ff267581253ae88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23de0469b8e597085dccb0d1c2b1d399002ff65d13c757f69ff267581253ae88/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3592033c3df3e86957ca9685c0078fcb41b78dbbf448ac9a0c6c339d66ebdb6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3592033c3df3e86957ca9685c0078fcb41b78dbbf448ac9a0c6c339d66ebdb6f/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42904b61dc9a39cabb621a14bbc9d5cba600731562df9d4d2db41f2f6c3376e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42904b61dc9a39cabb621a14bbc9d5cba600731562df9d4d2db41f2f6c3376e6/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46aeb6b2f93ef9c35817bf1c47f4294779f5b0719b7688c0d67a6f6a10f255ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46aeb6b2f93ef9c35817bf1c47f4294779f5b0719b7688c0d67a6f6a10f255ec/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4890a77387a12b66496d05e2b6945131e9bb41c66eb499151b9fd7d7cad91380/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4890a77387a12b66496d05e2b6945131e9bb41c66eb499151b9fd7d7cad91380/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5461af58acccb5b14fc3121dd8aae6be55ba422f1a9f1a14b7887cb42ef67497/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5461af58acccb5b14fc3121dd8aae6be55ba422f1a9f1a14b7887cb42ef67497/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73d983f21fc0e06c47c090cd39c024b8aa55e597bcc940eb514518a761a58b66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73d983f21fc0e06c47c090cd39c024b8aa55e597bcc940eb514518a761a58b66/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80b785777e04974bee3960968c36628f7ff27d00b92e31cbebf0503e391c371a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80b785777e04974bee3960968c36628f7ff27d00b92e31cbebf0503e391c371a/userdata/shm major:0 minor:308 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6792a1a69a64bd7eeb652ac9685ae29213a12dec8805ff6b29342ac559512dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6792a1a69a64bd7eeb652ac9685ae29213a12dec8805ff6b29342ac559512dc/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b31983de7934c38ad45951a23fdc243078200f7a6041a8464a46fbd921bc6713/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b31983de7934c38ad45951a23fdc243078200f7a6041a8464a46fbd921bc6713/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6022d71402a00d2c235b662c1ef924357fc899b05012dda33d9480c6e4fb1a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6022d71402a00d2c235b662c1ef924357fc899b05012dda33d9480c6e4fb1a5/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8da82b2c30c4cfd7f46f42b5d0b00779ba426d13aa7a797dbf2c2194b11dd0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8da82b2c30c4cfd7f46f42b5d0b00779ba426d13aa7a797dbf2c2194b11dd0c/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfe57c65270076254d8f4a998bb86c20193871b9e7dc1c741f66eccdb8c5c94f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfe57c65270076254d8f4a998bb86c20193871b9e7dc1c741f66eccdb8c5c94f/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8b340fa765bb9d414fb9355c0c7314ba243407539c8d5d6e309910061598c01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8b340fa765bb9d414fb9355c0c7314ba243407539c8d5d6e309910061598c01/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9/userdata/shm major:0 minor:266 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e838e5117fa1ca9f0e5d54ece4671b728057b3f2ec68515fa9e0090af5c1b0ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e838e5117fa1ca9f0e5d54ece4671b728057b3f2ec68515fa9e0090af5c1b0ce/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f/volumes/kubernetes.io~projected/kube-api-access-vkd4v:{mountpoint:/var/lib/kubelet/pods/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f/volumes/kubernetes.io~projected/kube-api-access-vkd4v major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26fe52bf-f29e-465a-b519-164d0f092c0b/volumes/kubernetes.io~projected/kube-api-access-nf46t:{mountpoint:/var/lib/kubelet/pods/26fe52bf-f29e-465a-b519-164d0f092c0b/volumes/kubernetes.io~projected/kube-api-access-nf46t major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/342d6281-4bdd-4090-ac8b-717ca2942a0a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/342d6281-4bdd-4090-ac8b-717ca2942a0a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~projected/kube-api-access-w6hb6:{mountpoint:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~projected/kube-api-access-w6hb6 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~projected/kube-api-access-2fwwk:{mountpoint:/var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~projected/kube-api-access-2fwwk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ecea087-8175-4789-a497-7beeb69e1610/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4ecea087-8175-4789-a497-7beeb69e1610/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/626fe428-7557-42b8-a473-0ab960c59898/volumes/kubernetes.io~projected/kube-api-access-d9gls:{mountpoint:/var/lib/kubelet/pods/626fe428-7557-42b8-a473-0ab960c59898/volumes/kubernetes.io~projected/kube-api-access-d9gls major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf/volumes/kubernetes.io~projected/kube-api-access-5xlkx:{mountpoint:/var/lib/kubelet/pods/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf/volumes/kubernetes.io~projected/kube-api-access-5xlkx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf/volumes/kubernetes.io~secret/metrics-certs major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76974834-71dc-4488-b393-ed2730c0276a/volumes/kubernetes.io~projected/kube-api-access-5qq5w:{mountpoint:/var/lib/kubelet/pods/76974834-71dc-4488-b393-ed2730c0276a/volumes/kubernetes.io~projected/kube-api-access-5qq5w major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~projected/kube-api-access-lw946:{mountpoint:/var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~projected/kube-api-access-lw946 major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/alertmanager/9 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~empty-dir/config-out major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~projected/kube-api-access-s2xn2:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~projected/kube-api-access-s2xn2 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~projected/tls-assets major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/config-volume major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/web-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~projected/kube-api-access-77f5l:{mountpoint:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~projected/kube-api-access-77f5l major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a533b40e-1b38-48c6-b409-d12ead6930a0/volumes/kubernetes.io~projected/kube-api-access-w6467:{mountpoint:/var/lib/kubelet/pods/a533b40e-1b38-48c6-b409-d12ead6930a0/volumes/kubernetes.io~projected/kube-api-access-w6467 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volumes/kubernetes.io~projected/kube-api-access-n7hdg:{mountpoint:/var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volumes/kubernetes.io~projected/kube-api-access-n7hdg major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb54163a-2ad1-4b02-93d6-f8eac877da15/volumes/kubernetes.io~projected/kube-api-access-dlxjg:{mountpoint:/var/lib/kubelet/pods/bb54163a-2ad1-4b02-93d6-f8eac877da15/volumes/kubernetes.io~projected/kube-api-access-dlxjg major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb54163a-2ad1-4b02-93d6-f8eac877da15/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/bb54163a-2ad1-4b02-93d6-f8eac877da15/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~projected/kube-api-access-ngrbz:{mountpoint:/var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~projected/kube-api-access-ngrbz major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d680dcf8-8cd0-488b-b8b1-b66368817faf/volumes/kubernetes.io~projected/kube-api-access-kzwgw:{mountpoint:/var/lib/kubelet/pods/d680dcf8-8cd0-488b-b8b1-b66368817faf/volumes/kubernetes.io~projected/kube-api-access-kzwgw major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d680dcf8-8cd0-488b-b8b1-b66368817faf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d680dcf8-8cd0-488b-b8b1-b66368817faf/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de01be03-1a51-4ada-9b3c-498276ad5677/volumes/kubernetes.io~projected/kube-api-access-dzw74:{mountpoint:/var/lib/kubelet/pods/de01be03-1a51-4ada-9b3c-498276ad5677/volumes/kubernetes.io~projected/kube-api-access-dzw74 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de01be03-1a51-4ada-9b3c-498276ad5677/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/de01be03-1a51-4ada-9b3c-498276ad5677/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e18c61c8-db8c-49ca-8fbd-9378be7e9c51/volumes/kubernetes.io~projected/kube-api-access-b5gg7:{mountpoint:/var/lib/kubelet/pods/e18c61c8-db8c-49ca-8fbd-9378be7e9c51/volumes/kubernetes.io~projected/kube-api-access-b5gg7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0eca957-f922-4106-8192-4d3d39e8a21d/volumes/kubernetes.io~projected/kube-api-access-hgfg5:{mountpoint:/var/lib/kubelet/pods/f0eca957-f922-4106-8192-4d3d39e8a21d/volumes/kubernetes.io~projected/kube-api-access-hgfg5 major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a833a8ab29fc6e1235814b7e3bb13c86ebda4bdbe64580e79065455153c9a1f7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e658c406bd85fb44c4603f3ecac0ba5affffd8dcf0a64c4c396a5acab00cf99a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e599e8a3bdf1bb9057381f141955eb1ea97fbe63c3e796d51d7edd043a7b8109/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/621d9ace6b04b0206a593b34aa42dadd388134c8ee25978169d03adc96e9c1d0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d7791d12975640b0ee2426ebcb7d7434b1bf10fc5014713f8bad528d63ba008d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/27f3891f9aac2e760ea4b65401ddc3e6fa133611811cc684b283f96f9dad2500/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c99a146ab7403e34690c7d104133597ffa50702cb93f562e554499221a965477/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3a62bc386665369da5b39dde72467868eeed33f9af942218935b3b0103f4a510/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2ccd7856728a077e49adc253fae3132f36350dfd39cb3fb4e1fbb03f56f76fe6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/4fe695580dce3af9fd177c05ead84485a9d05efb6b7be02d07410faae1c968cb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/584c32fc2d5ca362e3800301e8cc59fd20f41c1891b2641e5607e7d02be06b29/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/fb44feaa50acfe57779de07528d3fa9f039824140d62dd79aa83ae480072b49c/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/cf9a54a97b3945db46b2ceb78b55ef13ccba91e295af75cfae7ee988656884bf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/8b5507d4dd6b284e5a382a061afdc4710ebd2ce1c43bb9a2b408182600cd3397/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/72cad61e4f10f9cde8b753eb9c7a38702a2b0c853a8b470d4bfa630fec6b0bfa/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/adfd169076d389457fe69d9214532cf316c7014d101ed67e2337fa8e8b4c07c0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3664f5a3425cde5169326e43bf7905d8b029c6edd920aea26fd30309385bbb0b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e57fa1ea6a413f80c8098140b420dbc2fb2c2c12b42ea42ad6e15aef1109fa0b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/660727610e2385020c4e94c53c3066e44b0b29a150a0d4c587fb6d8924652abe/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1686ed19e9cb765f077bc966cf7f188695f0b3b483289ede7f6d25e8d4ba8dbe/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/36ed750d6e54840083ee6cf8802b9ea4b5a735663fe5a75c0b906f06da170298/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/3406d8f63ffab57af395256fcd04b2210f9c592dcab4d1c6c45d6bd6417ecbba/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/42f47e237e1842f73629881d0c3cdf21a09e2ca8af881427e4da5fbdfa68abb1/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/1614281d551aeb10b52e2ec16d7451f021d4531e1410f85c4b1198a90ee1b1aa/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/24927274a334c9b6918c71d7c5688d9877a16148ac6f19a653b785521a6f9f5d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/8c9bbbf2902a86fcd6983f1971879d75136342b2dab0a6c2341424f94f93d2a9/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/68e74034711e0ae890c8782f7e983b7e97923218d7840e905138931d882dff15/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/81f6b1bf72a84b83bb4cf2b88eeb241d8e9b3ca1e34ea3f22f851d83aa7321a2/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/83761665bf9c6109b1f273030ad69fd26d6dedcdb6ba2cb330c3e78756953320/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/e9fcac0a7f0eb7c273cf7e3cb3ce714b386df409971fb03dcc6a6c0b7eb85dc1/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/a532306db15965b2974a6cbc01bcee7d566dd189e7fdc1cd88379bfef69fd8ed/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1cb7742ba5d5b3518055eca3ef1acc72706cea11083d528d7b881f3366b8c6ab/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c89dd5a9f3b73c21b630504991f52e33704596de38796612139bfe55f6550d74/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/b083c74ccf9e420cdf6388d7ac1558851eb036a60209ae05de90b0d7174665de/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/9500d24a5510a792b96a0ffda8e2f3b5a161473a17de13842eeed461c268cb15/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/e013ff5b3b942ceded7b37a258b4ebeca299b361ded4630bd71cebd3bda42e44/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/717e390b6458d46fbf3ae37a22a318d6bf2a60eeb07fb640e2830e1c62257327/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/852410c37786b54ba1910ff21f37df5f7c5d11dce1d54c9b95ab57187bb2ac89/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6a9e3a99bcd45c8141ffee640d739f5283df7b1fdd29dc86567396f6e8971248/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/fcb2de2b6da743babdab0e8e62384a3bd2fb217bea7d2066a5323974b146c1e1/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/aa8b6f58005d75fa2d111a265560e406eb7eda0c58ff63add0c92d8f5a7ae899/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/80597b5bf6455164cebe4aab302ab59f2df405d31bbd1bb1472300f66ae41849/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/03a4ab290872c8e07e996b9a639308300b8d58d1d147d50998a1c18fb92887fc/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/aae7a674717ebc6402cbceab4bf7b16163080630162bda2b447e15b464f4f4cf/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/7dc0780a355e59d1228127e2974dbc74d65a51ca92dc5dc868e53a9183c11fea/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/9cb1ff4832950b3c0871b0f50859ce2c5efef542bf906dd7ed5aedd5076d5c24/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/826e638fc714b282405013c7c42df4d6c30ff2018c9a16c77fcbc7ee6fbf3041/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/72610c96ec1f4b5d88904483ce47a027acdf48b4cd45e8c334e7e06c8715a755/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/c68aa2701c47dadcc9875dcb6de93fa25f91acff9508560b8e0809c8fc4582bc/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/b9fb21212fa4ce7b79fd03023e28ff703bfbcf214912251e5ca1880a0ef72f18/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/9fc39ddc0600abd91ce793f60e60114620f31d19324d54004340fd461c033f2f/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/5fc6282084c520bfb15132f2d97be8b2ad090b1a2d9f68a943756ee15036506a/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/bc90b07bab7b2d88fabb6b2904ee8e560cd4de72309480d9d952608cd0cfaeff/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/c9883d33fb9d500f335f58e8560648327b28bad5322e8da58774bd0922ca65f9/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/5173442ebbcd988b06c8e47fe49c36e5a3f507ccceeddcf015e45c75a142d586/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/3ee4dbc13b89c4eff043c197668505a408b56a9533124d288347fef1b1e071d4/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/7f6b9a2c2d504425b71dcc9ff7894d4c31ec899a209a98856efa9b253e6095fd/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/aa68c58bae59efb41baded240f4382ae6e11c26666e7143455f800f90fcee472/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cfdf36d20f264afc1e5a6f466aed61174f74a26630d20e8b742d3c5bcaf56ba4/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/efea75f8575457fcde66f3cb7a33eb55635c3498b6facc24ff063bd6fa4984b0/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/d1cee3537f533171f7559ae481bf1d1e055e25a5cdee690afd2688fafc43cfe4/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/1191347172b31a3aeb6c4092b3595fce46b31c55fb7fec30e5ece2e494d3dc56/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a3f0582c64cbd4053267e1c402427f4a64ed2edb4581d6f1d0b6afea64c1cb33/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b23abe9258f793a121417c6269bba1575d0276fe24552d59995c636af59a095e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/b81f0c01f94fa885132230023e408d8a39c83142c839167e04ad6d0fc08ba348/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c268154e9552b3299a7585be4af1b845fbba07a8db0fb58fffffcbaec24ad972/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/56651db428d14a36d22e3e4e983ba66d5eed5be9b16c259fb944eb1e147ebd7e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 07:54:10.557230 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.556842    6430 manager.go:217] Machine: {Timestamp:2026-11-03 07:54:10.556375718 +0000 UTC m=+0.059211223 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100367 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2691129ce845463d813083a90a6b63 SystemUUID:ec269112-9ce8-4546-3d81-3083a90a6b63 BootID:d9667ff0-041e-41ab-82fd-e90a4b32b392 Filesystems:[{Device:/var/lib/kubelet/pods/d680dcf8-8cd0-488b-b8b1-b66368817faf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5461af58acccb5b14fc3121dd8aae6be55ba422f1a9f1a14b7887cb42ef67497/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~projected/kube-api-access-ngrbz DeviceMajor:0 DeviceMinor:240 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:315 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8da82b2c30c4cfd7f46f42b5d0b00779ba426d13aa7a797dbf2c2194b11dd0c/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:301 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:270 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:312 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23de0469b8e597085dccb0d1c2b1d399002ff65d13c757f69ff267581253ae88/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de01be03-1a51-4ada-9b3c-498276ad5677/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:272 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42904b61dc9a39cabb621a14bbc9d5cba600731562df9d4d2db41f2f6c3376e6/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d680dcf8-8cd0-488b-b8b1-b66368817faf/volumes/kubernetes.io~projected/kube-api-access-kzwgw DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~projected/kube-api-access-77f5l DeviceMajor:0 DeviceMinor:307 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~projected/kube-api-access-2fwwk DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f0eca957-f922-4106-8192-4d3d39e8a21d/volumes/kubernetes.io~projected/kube-api-access-hgfg5 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:313 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a533b40e-1b38-48c6-b409-d12ead6930a0/volumes/kubernetes.io~projected/kube-api-access-w6467 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:375 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:267 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:269 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:305 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f/volumes/kubernetes.io~projected/kube-api-access-vkd4v DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~projected/kube-api-access-w6hb6 DeviceMajor:0 DeviceMinor:316 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4ecea087-8175-4789-a497-7beeb69e1610/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:273 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4890a77387a12b66496d05e2b6945131e9bb41c66eb499151b9fd7d7cad91380/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e18c61c8-db8c-49ca-8fbd-9378be7e9c51/volumes/kubernetes.io~projected/kube-api-access-b5gg7 DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfe57c65270076254d8f4a998bb86c20193871b9e7dc1c741f66eccdb8c5c94f/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9/userdata/shm DeviceMajor:0 DeviceMinor:266 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26fe52bf-f29e-465a-b519-164d0f092c0b/volumes/kubernetes.io~projected/kube-api-access-nf46t DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~projected/kube-api-access-lw946 DeviceMajor:0 DeviceMinor:243 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de01be03-1a51-4ada-9b3c-498276ad5677/volumes/kubernetes.io~projected/kube-api-access-dzw74 DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e838e5117fa1ca9f0e5d54ece4671b728057b3f2ec68515fa9e0090af5c1b0ce/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:273 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/37e9ee43-9770-42c8-9064-92437099e68a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:314 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volumes/kubernetes.io~projected/kube-api-access-n7hdg DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:274 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a713597c05f9b478cf4d24047b99039ce83e00a81fd49d6d83dd52ec839128a/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bb54163a-2ad1-4b02-93d6-f8eac877da15/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:271 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb54163a-2ad1-4b02-93d6-f8eac877da15/volumes/kubernetes.io~projected/kube-api-access-dlxjg DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6792a1a69a64bd7eeb652ac9685ae29213a12dec8805ff6b29342ac559512dc/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9600bd0a-077f-40c5-b09d-eac5da61049b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:306 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/626fe428-7557-42b8-a473-0ab960c59898/volumes/kubernetes.io~projected/kube-api-access-d9gls DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6022d71402a00d2c235b662c1ef924357fc899b05012dda33d9480c6e4fb1a5/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73d983f21fc0e06c47c090cd39c024b8aa55e597bcc940eb514518a761a58b66/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/114512a799f2d7eb864d83a4aff0eefb65c2e5c3961aa714de039cc981fc883f/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/342d6281-4bdd-4090-ac8b-717ca2942a0a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:371 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80b785777e04974bee3960968c36628f7ff27d00b92e31cbebf0503e391c371a/userdata/shm DeviceMajor:0 DeviceMinor:308 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~projected/kube-api-access-s2xn2 DeviceMajor:0 DeviceMinor:276 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8b340fa765bb9d414fb9355c0c7314ba243407539c8d5d6e309910061598c01/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8387d126-b80b-43e1-978b-4ee863c4ddee/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:241 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b98521-a087-4c0d-a301-a49080ff65b5/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:248 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf/volumes/kubernetes.io~projected/kube-api-access-5xlkx DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76974834-71dc-4488-b393-ed2730c0276a/volumes/kubernetes.io~projected/kube-api-access-5qq5w DeviceMajor:0 DeviceMinor:380 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/079f77c5974971199aa139b9a43960bf5a69d77e69deebb7c4d26f5aab34a998/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b31983de7934c38ad45951a23fdc243078200f7a6041a8464a46fbd921bc6713/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:272 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46aeb6b2f93ef9c35817bf1c47f4294779f5b0719b7688c0d67a6f6a10f255ec/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab85a3f2-5d53-4d6a-bf1b-185db437abf1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3592033c3df3e86957ca9685c0078fcb41b78dbbf448ac9a0c6c339d66ebdb6f/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:268 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a713597c05f9b4 MacAddress:06:82:dc:30:0c:01 Speed:10000 Mtu:8901} {Name:4890a77387a12b6 MacAddress:36:47:73:8d:b1:fb Speed:10000 Mtu:8901} {Name:5461af58acccb5b MacAddress:f6:15:65:11:7f:1c Speed:10000 Mtu:8901} {Name:80b785777e04974 MacAddress:b6:6f:66:2b:02:9d Speed:10000 Mtu:8901} {Name:a6792a1a69a64bd MacAddress:0a:8c:25:9b:02:ed Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c9:3b:ab:27 Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:2f:04:fd:3a:7c Speed:0 Mtu:8901} {Name:c6022d71402a00d MacAddress:6e:66:1a:9d:ca:17 Speed:10000 Mtu:8901} {Name:c8da82b2c30c4cf MacAddress:2e:7b:8d:e0:ca:08 Speed:10000 Mtu:8901} {Name:d8b340fa765bb9d MacAddress:ca:8d:d2:37:86:31 Speed:10000 Mtu:8901} {Name:e28cd8256a9ac99 MacAddress:e2:40:22:fe:7a:30 Speed:10000 Mtu:8901} {Name:e838e5117fa1ca9 MacAddress:be:24:54:1b:6f:84 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c9:3b:ab:27 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d2:e4:5a:2f:ce:55 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:8e:7c:ca:a4:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 07:54:10.557421 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557235    6430 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 07:54:10.557421 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557283    6430 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 07:54:10.557624 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557601    6430 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 07:54:10.557750 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557625    6430 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-232.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 07:54:10.557794 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557753    6430 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 07:54:10.557794 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557760    6430 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 07:54:10.557794 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557774    6430 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:54:10.557794 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557792    6430 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:54:10.557899 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.557871    6430 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 07:54:10.558091 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.558079    6430 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 07:54:10.558206 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.558197    6430 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 07:54:10.558239 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.558234    6430 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 07:54:10.558270 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.558245    6430 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 07:54:10.558270 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.558254    6430 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 07:54:10.558270 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.558263    6430 kubelet.go:405] "Adding apiserver pod source"
Nov 03 07:54:10.558367 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.558271    6430 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 07:54:10.559191 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.559173    6430 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 07:54:10.559191 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.559189    6430 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:54:10.559760 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.559744    6430 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 07:54:10.562319 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562304    6430 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 07:54:10.562319 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562318    6430 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:54:10.562423 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562413    6430 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 07:54:10.562603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562593    6430 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 07:54:10.562633 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562622    6430 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 07:54:10.562713 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562704    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 07:54:10.562745 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562719    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 07:54:10.562745 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562726    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 07:54:10.562745 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562731    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 07:54:10.562745 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562736    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 07:54:10.562745 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562742    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562748    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562754    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562761    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562767    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562776    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562785    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562806    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 07:54:10.562872 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.562812    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 07:54:10.566102 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.566091    6430 server.go:1263] "Started kubelet"
Nov 03 07:54:10.566257 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.566230    6430 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 07:54:10.566311 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.566237    6430 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 07:54:10.566355 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.566326    6430 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 07:54:10.566615 ip-10-0-140-232 systemd[1]: Started Kubernetes Kubelet.
Nov 03 07:54:10.566938 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.566782    6430 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 07:54:10.569163 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.568672    6430 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 07:54:10.573273 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.573094    6430 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 07:54:10.573817 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.573752    6430 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.573804    6430 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:48:00 +0000 UTC" deadline="2028-07-12 18:06:35.935296726 +0000 UTC"
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.573846    6430 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14818h12m25.361453904s"
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.573886    6430 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.573897    6430 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.573968    6430 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:10.574040    6430 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-232.us-east-2.compute.internal\" not found"
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.575144    6430 factory.go:55] Registering systemd factory
Nov 03 07:54:10.575313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.575162    6430 factory.go:223] Registration of the systemd container factory successfully
Nov 03 07:54:10.575886 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.575782    6430 factory.go:153] Registering CRI-O factory
Nov 03 07:54:10.575886 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.575800    6430 factory.go:223] Registration of the crio container factory successfully
Nov 03 07:54:10.575999 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.575893    6430 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 07:54:10.575999 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.575913    6430 factory.go:103] Registering Raw factory
Nov 03 07:54:10.575999 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.575939    6430 manager.go:1199] Started watching for new ooms in manager
Nov 03 07:54:10.576372 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576355    6430 manager.go:319] Starting recovery of all containers
Nov 03 07:54:10.576598 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576552    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426baa2b-1ff2-4e15-bf7d-4efdcdf4b221" volumeName="kubernetes.io/projected/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-kube-api-access-2fwwk" seLinuxMountContext=""
Nov 03 07:54:10.576657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576604    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ecea087-8175-4789-a497-7beeb69e1610" volumeName="kubernetes.io/secret/4ecea087-8175-4789-a497-7beeb69e1610-agent-certs" seLinuxMountContext=""
Nov 03 07:54:10.576657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576622    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8387d126-b80b-43e1-978b-4ee863c4ddee" volumeName="kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-metrics-client-ca" seLinuxMountContext=""
Nov 03 07:54:10.576657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576637    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 07:54:10.576657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576653    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9600bd0a-077f-40c5-b09d-eac5da61049b" volumeName="kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-client-ca-bundle" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576665    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de01be03-1a51-4ada-9b3c-498276ad5677" volumeName="kubernetes.io/configmap/de01be03-1a51-4ada-9b3c-498276ad5677-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576677    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b98521-a087-4c0d-a301-a49080ff65b5" volumeName="kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576691    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e18c61c8-db8c-49ca-8fbd-9378be7e9c51" volumeName="kubernetes.io/projected/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-kube-api-access-b5gg7" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576707    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8387d126-b80b-43e1-978b-4ee863c4ddee" volumeName="kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576721    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9600bd0a-077f-40c5-b09d-eac5da61049b" volumeName="kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576731    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9600bd0a-077f-40c5-b09d-eac5da61049b" volumeName="kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576745    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a533b40e-1b38-48c6-b409-d12ead6930a0" volumeName="kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-cni-binary-copy" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576757    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b98521-a087-4c0d-a301-a49080ff65b5" volumeName="kubernetes.io/configmap/c8b98521-a087-4c0d-a301-a49080ff65b5-metrics-client-ca" seLinuxMountContext=""
Nov 03 07:54:10.576778 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576769    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de01be03-1a51-4ada-9b3c-498276ad5677" volumeName="kubernetes.io/empty-dir/de01be03-1a51-4ada-9b3c-498276ad5677-data-volume" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576816    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de01be03-1a51-4ada-9b3c-498276ad5677" volumeName="kubernetes.io/secret/de01be03-1a51-4ada-9b3c-498276ad5677-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576833    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-metrics-client-ca" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576864    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576883    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576899    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576911    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9600bd0a-077f-40c5-b09d-eac5da61049b" volumeName="kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576922    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576936    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576953    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576969    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a533b40e-1b38-48c6-b409-d12ead6930a0" volumeName="kubernetes.io/projected/a533b40e-1b38-48c6-b409-d12ead6930a0-kube-api-access-w6467" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576980    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab85a3f2-5d53-4d6a-bf1b-185db437abf1" volumeName="kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.576995    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="342d6281-4bdd-4090-ac8b-717ca2942a0a" volumeName="kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577008    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426baa2b-1ff2-4e15-bf7d-4efdcdf4b221" volumeName="kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-tmp" seLinuxMountContext=""
Nov 03 07:54:10.577038 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577022    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8387d126-b80b-43e1-978b-4ee863c4ddee" volumeName="kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-tls" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577052    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab85a3f2-5d53-4d6a-bf1b-185db437abf1" volumeName="kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-env-overrides" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577084    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb54163a-2ad1-4b02-93d6-f8eac877da15" volumeName="kubernetes.io/secret/bb54163a-2ad1-4b02-93d6-f8eac877da15-metrics-tls" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577097    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26fe52bf-f29e-465a-b519-164d0f092c0b" volumeName="kubernetes.io/projected/26fe52bf-f29e-465a-b519-164d0f092c0b-kube-api-access-nf46t" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577109    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9600bd0a-077f-40c5-b09d-eac5da61049b" volumeName="kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577123    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626fe428-7557-42b8-a473-0ab960c59898" volumeName="kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577139    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8387d126-b80b-43e1-978b-4ee863c4ddee" volumeName="kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577151    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab85a3f2-5d53-4d6a-bf1b-185db437abf1" volumeName="kubernetes.io/secret/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577164    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b98521-a087-4c0d-a301-a49080ff65b5" volumeName="kubernetes.io/projected/c8b98521-a087-4c0d-a301-a49080ff65b5-kube-api-access-ngrbz" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577176    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d680dcf8-8cd0-488b-b8b1-b66368817faf" volumeName="kubernetes.io/projected/d680dcf8-8cd0-488b-b8b1-b66368817faf-kube-api-access-kzwgw" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577187    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426baa2b-1ff2-4e15-bf7d-4efdcdf4b221" volumeName="kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-tuned" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577199    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626fe428-7557-42b8-a473-0ab960c59898" volumeName="kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577220    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577232    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab85a3f2-5d53-4d6a-bf1b-185db437abf1" volumeName="kubernetes.io/projected/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-kube-api-access-n7hdg" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577247    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0eca957-f922-4106-8192-4d3d39e8a21d" volumeName="kubernetes.io/projected/f0eca957-f922-4106-8192-4d3d39e8a21d-kube-api-access-hgfg5" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577260    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577273    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626fe428-7557-42b8-a473-0ab960c59898" volumeName="kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-binary-copy" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577284    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577298    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a533b40e-1b38-48c6-b409-d12ead6930a0" volumeName="kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-daemon-config" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577313    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf" volumeName="kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577328    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9600bd0a-077f-40c5-b09d-eac5da61049b" volumeName="kubernetes.io/empty-dir/9600bd0a-077f-40c5-b09d-eac5da61049b-audit-log" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577342    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-client-tls" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577354    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ecea087-8175-4789-a497-7beeb69e1610" volumeName="kubernetes.io/configmap/4ecea087-8175-4789-a497-7beeb69e1610-konnectivity-ca" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577366    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8387d126-b80b-43e1-978b-4ee863c4ddee" volumeName="kubernetes.io/empty-dir/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-textfile" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577379    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577394    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577406    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab85a3f2-5d53-4d6a-bf1b-185db437abf1" volumeName="kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-config" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577418    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b98521-a087-4c0d-a301-a49080ff65b5" volumeName="kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577430    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e18c61c8-db8c-49ca-8fbd-9378be7e9c51" volumeName="kubernetes.io/configmap/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-serviceca" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577469    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577485    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577497    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb54163a-2ad1-4b02-93d6-f8eac877da15" volumeName="kubernetes.io/configmap/bb54163a-2ad1-4b02-93d6-f8eac877da15-config-volume" seLinuxMountContext=""
Nov 03 07:54:10.577507 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577513    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb54163a-2ad1-4b02-93d6-f8eac877da15" volumeName="kubernetes.io/empty-dir/bb54163a-2ad1-4b02-93d6-f8eac877da15-tmp-dir" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577526    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d680dcf8-8cd0-488b-b8b1-b66368817faf" volumeName="kubernetes.io/secret/d680dcf8-8cd0-488b-b8b1-b66368817faf-cert" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577543    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0eca957-f922-4106-8192-4d3d39e8a21d" volumeName="kubernetes.io/configmap/f0eca957-f922-4106-8192-4d3d39e8a21d-iptables-alerter-script" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577556    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577570    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf" volumeName="kubernetes.io/projected/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-kube-api-access-5xlkx" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577583    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577598    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16c1f0f2-b9b6-4659-97fb-9b3092e04f5f" volumeName="kubernetes.io/empty-dir/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-tmp-dir" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577610    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16c1f0f2-b9b6-4659-97fb-9b3092e04f5f" volumeName="kubernetes.io/projected/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-kube-api-access-vkd4v" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577627    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/projected/37e9ee43-9770-42c8-9064-92437099e68a-kube-api-access-w6hb6" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577639    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626fe428-7557-42b8-a473-0ab960c59898" volumeName="kubernetes.io/projected/626fe428-7557-42b8-a473-0ab960c59898-kube-api-access-d9gls" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577650    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8387d126-b80b-43e1-978b-4ee863c4ddee" volumeName="kubernetes.io/projected/8387d126-b80b-43e1-978b-4ee863c4ddee-kube-api-access-lw946" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577662    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9600bd0a-077f-40c5-b09d-eac5da61049b" volumeName="kubernetes.io/projected/9600bd0a-077f-40c5-b09d-eac5da61049b-kube-api-access-77f5l" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577675    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de01be03-1a51-4ada-9b3c-498276ad5677" volumeName="kubernetes.io/projected/de01be03-1a51-4ada-9b3c-498276ad5677-kube-api-access-dzw74" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577687    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37e9ee43-9770-42c8-9064-92437099e68a" volumeName="kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-federate-client-tls" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577700    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76974834-71dc-4488-b393-ed2730c0276a" volumeName="kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577717    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84caeca2-05a1-4588-b70e-c85607c1d219" volumeName="kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577735    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb54163a-2ad1-4b02-93d6-f8eac877da15" volumeName="kubernetes.io/projected/bb54163a-2ad1-4b02-93d6-f8eac877da15-kube-api-access-dlxjg" seLinuxMountContext=""
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577747    6430 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577753    6430 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.577883    6430 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:54:10.578428 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.578321    6430 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:54:10.580121 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.580102    6430 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:54:10.584086 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.582545    6430 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 07:54:10.607773 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.607726    6430 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 07:54:10.609526 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.609382    6430 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 07:54:10.609526 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.609405    6430 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 07:54:10.609526 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.609425    6430 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 07:54:10.609526 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:10.609466    6430 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 07:54:10.613074 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.613037    6430 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:54:10.635475 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.635426    6430 generic.go:356] "Generic (PLEG): container finished" podID="8387d126-b80b-43e1-978b-4ee863c4ddee" containerID="d3f20f54104583c7677d08e014fa8f488539570d92544fb5c1a275627be3cace" exitCode=0
Nov 03 07:54:10.647361 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.647336    6430 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="05483ec690fd98782493b215fefb1387db43a011e3c7dbd9c5813802a448b15c" exitCode=0
Nov 03 07:54:10.647361 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.647358    6430 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="427746251aa2f82f883ca20911cdee35513ed1388e34f320bcca3412fdeb00e8" exitCode=0
Nov 03 07:54:10.647361 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.647365    6430 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="f6cc3519587c7f62a38ce7e491eef0f177888e9630e584d9e2d8a5f9f5c59946" exitCode=0
Nov 03 07:54:10.647558 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.647371    6430 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="d6acda9d6ab18c438a5c460c5d50f6b0075e50174e3ebc30b02097055b4538e8" exitCode=0
Nov 03 07:54:10.647558 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.647378    6430 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="3c1ec5884f86d8b3fb4adba0c1bb4d68021477546f0508106609ab39f5897f5f" exitCode=0
Nov 03 07:54:10.647558 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.647385    6430 generic.go:356] "Generic (PLEG): container finished" podID="626fe428-7557-42b8-a473-0ab960c59898" containerID="4da97e28241da3da8e95f5447dece0428ff4d9803022c0fd3e92e66f256dde76" exitCode=0
Nov 03 07:54:10.653434 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.653406    6430 generic.go:356] "Generic (PLEG): container finished" podID="a15f1c78c19d0b39887f68042df09ebe" containerID="fe21e5fd76dbbbc045d498cc53a21f0ef9441bdffe61119fcef9ca48d0a6884e" exitCode=0
Nov 03 07:54:10.659142 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.659118    6430 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700" exitCode=0
Nov 03 07:54:10.679699 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.679680    6430 manager.go:324] Recovery completed
Nov 03 07:54:10.698792 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698737    6430 cpu_manager.go:226] "Starting" policy="none"
Nov 03 07:54:10.698792 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698751    6430 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 07:54:10.698792 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698766    6430 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 07:54:10.698946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698889    6430 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 07:54:10.698946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698899    6430 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 07:54:10.698946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698916    6430 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 07:54:10.698946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698921    6430 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 07:54:10.698946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698926    6430 policy_none.go:50] "Start"
Nov 03 07:54:10.698946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698934    6430 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 07:54:10.698946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.698942    6430 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 07:54:10.699982 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.699970    6430 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 07:54:10.699982 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.699983    6430 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 07:54:10.700045 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.699989    6430 policy_none.go:44] "Start"
Nov 03 07:54:10.704220 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.704196    6430 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 07:54:10.706589 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.706571    6430 manager.go:341] "Starting Device Plugin manager"
Nov 03 07:54:10.707221 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:10.706595    6430 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 07:54:10.707283 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707224    6430 server.go:87] "Starting device plugin registration server"
Nov 03 07:54:10.707471 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707460    6430 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 07:54:10.707503 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707474    6430 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 07:54:10.707563 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707544    6430 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 07:54:10.707678 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707645    6430 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 07:54:10.707743 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707680    6430 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 07:54:10.707743 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707687    6430 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 07:54:10.707839 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.707738    6430 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T07:54:10.707671088Z","UUID":"627df190-1e3a-4962-bb8d-3abd11133360","Handler":null,"Name":"","Endpoint":""}
Nov 03 07:54:10.708989 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.708969    6430 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 07:54:10.709085 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.709000    6430 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 07:54:10.709938 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:10.709533    6430 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 07:54:10.709938 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.709541    6430 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"]
Nov 03 07:54:10.712939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.711159    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"40f16205502b8d2684b7e492677380c9","Type":"ContainerStarted","Data":"5eb743171082d7f75f0f428382796a40fc0c63fa860d6911e19f603296d007b5"}
Nov 03 07:54:10.712939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.711231    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"40f16205502b8d2684b7e492677380c9","Type":"ContainerStarted","Data":"e6dee9577a5389be3513e7774107b6906234a75b4e81cd332e1c25e3f2257aa1"}
Nov 03 07:54:10.712939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.711254    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"a15f1c78c19d0b39887f68042df09ebe","Type":"ContainerStarted","Data":"91c435a3d5b3333b6ef406823f9dc8d56b28ff42bb88936102441034d84bf0de"}
Nov 03 07:54:10.712939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.711281    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"a15f1c78c19d0b39887f68042df09ebe","Type":"ContainerDied","Data":"fe21e5fd76dbbbc045d498cc53a21f0ef9441bdffe61119fcef9ca48d0a6884e"}
Nov 03 07:54:10.712939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.711302    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal" event={"ID":"a15f1c78c19d0b39887f68042df09ebe","Type":"ContainerStarted","Data":"7f1b7b7204b6fb5050efd5d6fb8e0b064773feca2398922bf34efe8d746ea15e"}
Nov 03 07:54:10.712939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.711430    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.712939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.711474    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.724043 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.724023    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:54:10.724147 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:10.724085    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.724472 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.724456    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:54:10.724554 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:10.724497    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.783440 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.783416    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.783526 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.783444    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.783526 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.783461    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f16205502b8d2684b7e492677380c9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"40f16205502b8d2684b7e492677380c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.808531 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.808510    6430 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:54:10.810484 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.810469    6430 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:54:10.810575 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.810500    6430 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:54:10.810575 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.810515    6430 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-232.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:54:10.810575 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.810569    6430 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.822291 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.822276    6430 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.822333 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.822321    6430 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.884211 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.884185    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.884309 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.884218    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.884309 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.884243    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f16205502b8d2684b7e492677380c9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"40f16205502b8d2684b7e492677380c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.884309 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.884281    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.884309 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.884294    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40f16205502b8d2684b7e492677380c9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"40f16205502b8d2684b7e492677380c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:10.884309 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:10.884293    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15f1c78c19d0b39887f68042df09ebe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" (UID: \"a15f1c78c19d0b39887f68042df09ebe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:11.558957 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.558923    6430 apiserver.go:52] "Watching apiserver"
Nov 03 07:54:11.567383 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.567355    6430 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 07:54:11.568455 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.568424    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4p5vg","openshift-ingress-canary/ingress-canary-45pld","openshift-monitoring/node-exporter-vfxjv","openshift-insights/insights-runtime-extractor-lfvbs","openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2","openshift-ovn-kubernetes/ovnkube-node-dgl7z","openshift-multus/multus-additional-cni-plugins-6tqbr","openshift-network-operator/iptables-alerter-cknqz","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-898qp","openshift-multus/multus-f6vt5","openshift-image-registry/node-ca-srvsw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal","openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l","kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-m7nbf","openshift-network-diagnostics/network-check-target-k5mr4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn","openshift-monitoring/metrics-server-688749c497-5dxnm","kube-system/konnectivity-agent-254xf","openshift-cluster-node-tuning-operator/tuned-f5b6x","openshift-dns/node-resolver-j7zvx"]
Nov 03 07:54:11.581579 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.581560    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.582039 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582017    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 07:54:11.582039 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582030    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 07:54:11.582039 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582017    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.582243 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582039    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 07:54:11.582243 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582119    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.582310 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582246    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.582338 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582307    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 07:54:11.582338 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582311    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.582394 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582355    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.582394 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582391    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.582591 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582577    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 07:54:11.582688 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582671    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 07:54:11.582688 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582685    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 07:54:11.582794 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582709    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.582794 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582748    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.582852 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582819    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 07:54:11.582852 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582827    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.582852 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582847    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 07:54:11.582927 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582876    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 07:54:11.582927 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582908    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 07:54:11.582927 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582919    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582928    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpwxk\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582961    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582970    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-thqm7\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582976    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582977    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582996    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583002    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 07:54:11.583007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583005    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.582911    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583023    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583029    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583032    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583074    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4xzk\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583076    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dc6md\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583088    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.583260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583108    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.583563 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583419    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.583634 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.583617    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.584128 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.584105    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.584247 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.584232    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 07:54:11.585219 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.585199    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 07:54:11.585308 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.585241    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 07:54:11.585904 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.585884    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 07:54:11.586717 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.586699    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 07:54:11.586830 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.586817    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 07:54:11.586918 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.586906    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pdlxh\""
Nov 03 07:54:11.586966 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.586939    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 07:54:11.587009 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.586972    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 07:54:11.587097 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587084    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c2t62\""
Nov 03 07:54:11.587249 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587233    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 07:54:11.587319 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587264    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwf4\""
Nov 03 07:54:11.587319 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587309    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-conf-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.587405 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587341    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.587405 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587347    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.587405 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587347    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-run\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.587483 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587426    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 07:54:11.587483 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587445    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-tmp\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.587483 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587462    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\""
Nov 03 07:54:11.587656 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587483    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 07:54:11.587656 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587534    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bmmvq\""
Nov 03 07:54:11.587656 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587539    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 07:54:11.587656 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587530    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-socket-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.587656 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587591    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\""
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587618    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587721    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587715    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-tmp\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587630    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-netns\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587741    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\""
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587775    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-etc-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587783    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587795    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wvjx7\""
Nov 03 07:54:11.587837 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587799    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-cnibin\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587832    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d835iev2t6ar\""
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587854    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjbhz\""
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587844    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-modprobe-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587906    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587908    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587912    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-device-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587952    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ghck6\""
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587955    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf46t\" (UniqueName: \"kubernetes.io/projected/26fe52bf-f29e-465a-b519-164d0f092c0b-kube-api-access-nf46t\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.587977    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-ovn\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588004    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588015    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de01be03-1a51-4ada-9b3c-498276ad5677-crio-socket\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588084    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5spf6\""
Nov 03 07:54:11.588134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588106    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 07:54:11.588478 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588186    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 07:54:11.588478 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588248    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 07:54:11.588669 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588111    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.588703 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588691    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-dbus\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.588755 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588727    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkd4v\" (UniqueName: \"kubernetes.io/projected/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-kube-api-access-vkd4v\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:54:11.588755 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588750    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588755    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-sys\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588751    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2mmmx\""
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588790    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9glt\""
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588795    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588788    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-slash\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588788    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8t4tj\""
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588733    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-config\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.588850 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588751    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588876    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c4vhz\""
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588884    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-env-overrides\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588903    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.588958    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589015    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de01be03-1a51-4ada-9b3c-498276ad5677-data-volume\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589082    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-multus-certs\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589111    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysconfig\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589115    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4jdjw\""
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589140    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-config\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-tuned\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589175    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de01be03-1a51-4ada-9b3c-498276ad5677-data-volume\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.589248 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589235    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-env-overrides\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589243    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxjg\" (UniqueName: \"kubernetes.io/projected/bb54163a-2ad1-4b02-93d6-f8eac877da15-kube-api-access-dlxjg\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589288    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-tuned\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589294    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-systemd-units\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589310    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589317    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589345    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzw74\" (UniqueName: \"kubernetes.io/projected/de01be03-1a51-4ada-9b3c-498276ad5677-kube-api-access-dzw74\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589382    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-kubelet\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589409    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-systemd\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589432    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-conf\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589496    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589535    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfg5\" (UniqueName: \"kubernetes.io/projected/f0eca957-f922-4106-8192-4d3d39e8a21d-kube-api-access-hgfg5\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589561    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwgw\" (UniqueName: \"kubernetes.io/projected/d680dcf8-8cd0-488b-b8b1-b66368817faf-kube-api-access-kzwgw\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589623    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-k8s-cni-cncf-io\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589644    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-bin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589659    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-etc-kubernetes\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589759    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589776    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwwk\" (UniqueName: \"kubernetes.io/projected/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-kube-api-access-2fwwk\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589791    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb54163a-2ad1-4b02-93d6-f8eac877da15-tmp-dir\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589841    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb54163a-2ad1-4b02-93d6-f8eac877da15-tmp-dir\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589861    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-kubelet\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589876    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0eca957-f922-4106-8192-4d3d39e8a21d-host-slash\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589893    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6467\" (UniqueName: \"kubernetes.io/projected/a533b40e-1b38-48c6-b409-d12ead6930a0-kube-api-access-w6467\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.589924    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590000    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590080    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/342d6281-4bdd-4090-ac8b-717ca2942a0a-original-pull-secret\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590093    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ecea087-8175-4789-a497-7beeb69e1610-agent-certs\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590126    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590137    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-host\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590169    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-var-lib-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590188    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7hdg\" (UniqueName: \"kubernetes.io/projected/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-kube-api-access-n7hdg\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590203    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-hostroot\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590230    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590239    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590248    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-tmp-dir\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590303    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ecea087-8175-4789-a497-7beeb69e1610-agent-certs\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590348    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-registration-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590367    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-script-lib\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590427    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-tmp-dir\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590424    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d680dcf8-8cd0-488b-b8b1-b66368817faf-cert\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:54:11.590515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590519    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-metrics-certs\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590538    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-system-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590565    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-cnibin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590588    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovnkube-script-lib\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590589    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-kubernetes\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590601    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d680dcf8-8cd0-488b-b8b1-b66368817faf-cert\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590617    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-lib-modules\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590662    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-sys-fs\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590687    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54163a-2ad1-4b02-93d6-f8eac877da15-config-volume\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590710    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590773    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-log-socket\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590802    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-socket-dir-parent\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590823    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54163a-2ad1-4b02-93d6-f8eac877da15-config-volume\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590830    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-daemon-config\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590853    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590916    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-systemd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590950    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-bin\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-netd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.590993    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gls\" (UniqueName: \"kubernetes.io/projected/626fe428-7557-42b8-a473-0ab960c59898-kube-api-access-d9gls\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591012    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlkx\" (UniqueName: \"kubernetes.io/projected/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-kube-api-access-5xlkx\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591020    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-daemon-config\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591093    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-cni-binary-copy\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591175    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-multus\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591259    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-kubelet-config\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591275    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-hosts-file\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591291    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a533b40e-1b38-48c6-b409-d12ead6930a0-cni-binary-copy\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591301    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-host\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591404    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-node-log\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.591426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591434    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ecea087-8175-4789-a497-7beeb69e1610-konnectivity-ca\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591449    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5gg7\" (UniqueName: \"kubernetes.io/projected/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-kube-api-access-b5gg7\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591465    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591482    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0eca957-f922-4106-8192-4d3d39e8a21d-iptables-alerter-script\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591529    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de01be03-1a51-4ada-9b3c-498276ad5677-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591551    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de01be03-1a51-4ada-9b3c-498276ad5677-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591582    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-netns\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591607    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591626    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ecea087-8175-4789-a497-7beeb69e1610-konnectivity-ca\") pod \"konnectivity-agent-254xf\" (UID: \"4ecea087-8175-4789-a497-7beeb69e1610\") " pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591630    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-system-cni-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591681    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591707    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0eca957-f922-4106-8192-4d3d39e8a21d-iptables-alerter-script\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591722    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-serviceca\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591770    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de01be03-1a51-4ada-9b3c-498276ad5677-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591850    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/626fe428-7557-42b8-a473-0ab960c59898-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591885    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591902    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591940    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-var-lib-kubelet\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591971    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb54163a-2ad1-4b02-93d6-f8eac877da15-metrics-tls\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591997    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovn-node-metrics-cert\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.591998    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de01be03-1a51-4ada-9b3c-498276ad5677-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.592160    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb54163a-2ad1-4b02-93d6-f8eac877da15-metrics-tls\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.592247    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-ovn-node-metrics-cert\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.592504 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.592317    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-serviceca\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:54:11.593941 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.593923    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf46t\" (UniqueName: \"kubernetes.io/projected/26fe52bf-f29e-465a-b519-164d0f092c0b-kube-api-access-nf46t\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.594293 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.594277    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkd4v\" (UniqueName: \"kubernetes.io/projected/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-kube-api-access-vkd4v\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:54:11.594831 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.594811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxjg\" (UniqueName: \"kubernetes.io/projected/bb54163a-2ad1-4b02-93d6-f8eac877da15-kube-api-access-dlxjg\") pod \"dns-default-4p5vg\" (UID: \"bb54163a-2ad1-4b02-93d6-f8eac877da15\") " pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:11.595287 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.595268    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwgw\" (UniqueName: \"kubernetes.io/projected/d680dcf8-8cd0-488b-b8b1-b66368817faf-kube-api-access-kzwgw\") pod \"ingress-canary-45pld\" (UID: \"d680dcf8-8cd0-488b-b8b1-b66368817faf\") " pod="openshift-ingress-canary/ingress-canary-45pld"
Nov 03 07:54:11.595371 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.595351    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwwk\" (UniqueName: \"kubernetes.io/projected/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-kube-api-access-2fwwk\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.595439 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.595426    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzw74\" (UniqueName: \"kubernetes.io/projected/de01be03-1a51-4ada-9b3c-498276ad5677-kube-api-access-dzw74\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.595546 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.595532    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6467\" (UniqueName: \"kubernetes.io/projected/a533b40e-1b38-48c6-b409-d12ead6930a0-kube-api-access-w6467\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.595698 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.595683    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfg5\" (UniqueName: \"kubernetes.io/projected/f0eca957-f922-4106-8192-4d3d39e8a21d-kube-api-access-hgfg5\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:54:11.597722 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.597702    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlkx\" (UniqueName: \"kubernetes.io/projected/6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf-kube-api-access-5xlkx\") pod \"network-metrics-daemon-m7nbf\" (UID: \"6c5d72b7-9d7e-42b2-9a2e-8c532c7522cf\") " pod="openshift-multus/network-metrics-daemon-m7nbf"
Nov 03 07:54:11.597861 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.597844    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5gg7\" (UniqueName: \"kubernetes.io/projected/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-kube-api-access-b5gg7\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:54:11.597952 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.597900    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gls\" (UniqueName: \"kubernetes.io/projected/626fe428-7557-42b8-a473-0ab960c59898-kube-api-access-d9gls\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.598629 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.598610    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7hdg\" (UniqueName: \"kubernetes.io/projected/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-kube-api-access-n7hdg\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.598701 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.598657    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq5w\" (UniqueName: \"kubernetes.io/projected/76974834-71dc-4488-b393-ed2730c0276a-kube-api-access-5qq5w\") pod \"network-check-target-k5mr4\" (UID: \"76974834-71dc-4488-b393-ed2730c0276a\") " pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:11.666472 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.666440    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:11.666620 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.666589    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:11.675657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.675626    6430 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 07:54:11.676354 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.676337    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:54:11.676427 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:11.676382    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:11.676712 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.676696    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:54:11.676786 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:11.676736    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-232.us-east-2.compute.internal"
Nov 03 07:54:11.692386 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692363    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.692491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692393    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-socket-dir-parent\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.692491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692410    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.692491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692424    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-systemd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.692491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692440    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-multus\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.692491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692458    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-node-log\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.692491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692480    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692498    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-socket-dir-parent\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692515    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-multus\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692504    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-systemd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692508    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692506    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692556    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-node-log\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692586    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-netns\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692614    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-root\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692619    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-netns\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692578    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692634    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692656    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-metrics-client-ca\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692675    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692696    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hb6\" (UniqueName: \"kubernetes.io/projected/37e9ee43-9770-42c8-9064-92437099e68a-kube-api-access-w6hb6\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692720    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9600bd0a-077f-40c5-b09d-eac5da61049b-audit-log\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.692730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692731    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692743    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692770    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692797    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692812    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9600bd0a-077f-40c5-b09d-eac5da61049b-audit-log\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692823    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-metrics-server-audit-profiles\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692834    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692857    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-socket-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692870    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-metrics-client-ca\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692883    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-cnibin\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692910    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-modprobe-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692943    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-socket-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692961    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-device-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692989    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de01be03-1a51-4ada-9b3c-498276ad5677-crio-socket\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693008    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-device-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.692947    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693016    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-client-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693075    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-dbus\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693088    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de01be03-1a51-4ada-9b3c-498276ad5677-crio-socket\") pod \"insights-runtime-extractor-lfvbs\" (UID: \"de01be03-1a51-4ada-9b3c-498276ad5677\") " pod="openshift-insights/insights-runtime-extractor-lfvbs"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693095    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693100    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-modprobe-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693100    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-sys\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693133    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693142    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-sys\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693152    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-cnibin\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693143    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-slash\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693190    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-slash\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693191    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-federate-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693154    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-dbus\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693140    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-metrics-server-audit-profiles\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693231    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-multus-certs\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693233    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-client-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693252    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysconfig\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693270    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-sys\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693285    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw946\" (UniqueName: \"kubernetes.io/projected/8387d126-b80b-43e1-978b-4ee863c4ddee-kube-api-access-lw946\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693299    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-conf\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693326    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysconfig\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693340    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-multus-certs\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-federate-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693339    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-tls\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693404    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-k8s-cni-cncf-io\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.693413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693424    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-etc-kubernetes\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693445    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0eca957-f922-4106-8192-4d3d39e8a21d-host-slash\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693468    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77f5l\" (UniqueName: \"kubernetes.io/projected/9600bd0a-077f-40c5-b09d-eac5da61049b-kube-api-access-77f5l\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693482    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-run-k8s-cni-cncf-io\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693488    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-conf\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693494    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693520    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0eca957-f922-4106-8192-4d3d39e8a21d-host-slash\") pod \"iptables-alerter-cknqz\" (UID: \"f0eca957-f922-4106-8192-4d3d39e8a21d\") " pod="openshift-network-operator/iptables-alerter-cknqz"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693526    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-tls\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693523    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693532    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-etc-kubernetes\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693621    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-host\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-var-lib-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693672    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-hostroot\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693690    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-serving-certs-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693709    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693708    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8b98521-a087-4c0d-a301-a49080ff65b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693725    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-var-lib-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693715    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-tls\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693764    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-system-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693771    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-host\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693789    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-sys-fs\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693828    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-hostroot\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693859    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-log-socket\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693866    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-system-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693883    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-netd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693887    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-serving-certs-ca-bundle\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693907    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693918    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-tls\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693920    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-sys-fs\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693949    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-netd\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693950    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-log-socket\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693974    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.693973    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694012    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694041    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-kubelet-config\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694084    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-hosts-file\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694110    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-host\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694120    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9600bd0a-077f-40c5-b09d-eac5da61049b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694134    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-textfile\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694159    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-client-certs\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694173    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/342d6281-4bdd-4090-ac8b-717ca2942a0a-kubelet-config\") pod \"global-pull-secret-syncer-898qp\" (UID: \"342d6281-4bdd-4090-ac8b-717ca2942a0a\") " pod="kube-system/global-pull-secret-syncer-898qp"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694177    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16c1f0f2-b9b6-4659-97fb-9b3092e04f5f-hosts-file\") pod \"node-resolver-j7zvx\" (UID: \"16c1f0f2-b9b6-4659-97fb-9b3092e04f5f\") " pod="openshift-dns/node-resolver-j7zvx"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694187    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694212    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-metrics-client-ca\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694237    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-system-cni-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694239    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-textfile\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694189    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e18c61c8-db8c-49ca-8fbd-9378be7e9c51-host\") pod \"node-ca-srvsw\" (UID: \"e18c61c8-db8c-49ca-8fbd-9378be7e9c51\") " pod="openshift-image-registry/node-ca-srvsw"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694263    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694286    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-var-lib-kubelet\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694313    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-wtmp\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694339    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694351    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-telemeter-client-tls\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694367    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-conf-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694381    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9600bd0a-077f-40c5-b09d-eac5da61049b-secret-metrics-server-client-certs\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694392    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-run\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694417    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-netns\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694426    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694443    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-etc-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694467    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngrbz\" (UniqueName: \"kubernetes.io/projected/c8b98521-a087-4c0d-a301-a49080ff65b5-kube-api-access-ngrbz\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694495    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/626fe428-7557-42b8-a473-0ab960c59898-system-cni-dir\") pod \"multus-additional-cni-plugins-6tqbr\" (UID: \"626fe428-7557-42b8-a473-0ab960c59898\") " pod="openshift-multus/multus-additional-cni-plugins-6tqbr"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694767    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-ovn\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694863    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694896    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694930    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-var-lib-kubelet\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694932    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xn2\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.694988    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695031    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-systemd-units\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695039    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695086    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695108    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695115    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695133    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-cni-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695182    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-run-netns\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695237    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-kubelet\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695271    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-systemd\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.695277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695297    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695327    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37e9ee43-9770-42c8-9064-92437099e68a-metrics-client-ca\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695347    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695374    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695395    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-sysctl-d\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695397    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-bin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695434    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-systemd-units\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-multus-conf-dir\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695481    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-cni-bin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695492    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37e9ee43-9770-42c8-9064-92437099e68a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695556    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-systemd\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695574    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-run\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695574    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695495    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-etc-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695621    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-kubelet\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695773    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-ovn\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695854    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695889    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695923    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-host-var-lib-kubelet\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695933    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-kubelet\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695967    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.695991    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-registration-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696031    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b98521-a087-4c0d-a301-a49080ff65b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696043    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26fe52bf-f29e-465a-b519-164d0f092c0b-registration-dir\") pod \"aws-ebs-csi-driver-node-qsfdn\" (UID: \"26fe52bf-f29e-465a-b519-164d0f092c0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsfdn"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696052    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-accelerators-collector-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696122    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696246    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696270    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b98521-a087-4c0d-a301-a49080ff65b5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696299    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-accelerators-collector-config\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696307    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696355    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-cnibin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696383    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-kubernetes\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696416    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-lib-modules\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696447    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696463    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a533b40e-1b38-48c6-b409-d12ead6930a0-cnibin\") pod \"multus-f6vt5\" (UID: \"a533b40e-1b38-48c6-b409-d12ead6930a0\") " pod="openshift-multus/multus-f6vt5"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696467    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-etc-kubernetes\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696480    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-bin\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696546    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-host-cni-bin\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696559    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/426baa2b-1ff2-4e15-bf7d-4efdcdf4b221-lib-modules\") pod \"tuned-f5b6x\" (UID: \"426baa2b-1ff2-4e15-bf7d-4efdcdf4b221\") " pod="openshift-cluster-node-tuning-operator/tuned-f5b6x"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696594    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab85a3f2-5d53-4d6a-bf1b-185db437abf1-run-openvswitch\") pod \"ovnkube-node-dgl7z\" (UID: \"ab85a3f2-5d53-4d6a-bf1b-185db437abf1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696720    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.696926    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.698280    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw946\" (UniqueName: \"kubernetes.io/projected/8387d126-b80b-43e1-978b-4ee863c4ddee-kube-api-access-lw946\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.698366 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.698304    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hb6\" (UniqueName: \"kubernetes.io/projected/37e9ee43-9770-42c8-9064-92437099e68a-kube-api-access-w6hb6\") pod \"telemeter-client-66b6b9b7d5-vw49l\" (UID: \"37e9ee43-9770-42c8-9064-92437099e68a\") " pod="openshift-monitoring/telemeter-client-66b6b9b7d5-vw49l"
Nov 03 07:54:11.700236 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.698967    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77f5l\" (UniqueName: \"kubernetes.io/projected/9600bd0a-077f-40c5-b09d-eac5da61049b-kube-api-access-77f5l\") pod \"metrics-server-688749c497-5dxnm\" (UID: \"9600bd0a-077f-40c5-b09d-eac5da61049b\") " pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:11.700652 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.700370    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngrbz\" (UniqueName: \"kubernetes.io/projected/c8b98521-a087-4c0d-a301-a49080ff65b5-kube-api-access-ngrbz\") pod \"openshift-state-metrics-7bd7489957-98nq2\" (UID: \"c8b98521-a087-4c0d-a301-a49080ff65b5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-98nq2"
Nov 03 07:54:11.700830 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.700811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.700938 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.700919    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xn2\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\") pod \"alertmanager-main-0\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:11.797738 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.797702    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-wtmp\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.797871 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.797764    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-root\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.797871 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.797800    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-sys\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.797871 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.797830    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-node-exporter-wtmp\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.797981 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.797900    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-root\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.797981 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.797905    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8387d126-b80b-43e1-978b-4ee863c4ddee-sys\") pod \"node-exporter-vfxjv\" (UID: \"8387d126-b80b-43e1-978b-4ee863c4ddee\") " pod="openshift-monitoring/node-exporter-vfxjv"
Nov 03 07:54:11.897474 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:11.897409    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:12.004932 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:12.004900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:13.209635 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:13.209604    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4p5vg"
Nov 03 07:54:14.552782 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:14.552750    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:14.908797 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:14.908712    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:15.015395 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:15.015363    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:15.176236 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:15.176173    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:16.975598 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:16.975568    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-688749c497-5dxnm"
Nov 03 07:54:19.138194 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:19.138150    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:19.776567 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:19.776535    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:19.776751 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:19.776737    6430 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:54:21.017317 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:21.017286    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:21.124794 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:21.124758    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:21.226275 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:21.226247    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:28.373521 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:28.373479    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:54:30.535466 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:30.535442    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dgl7z"
Nov 03 07:54:31.744809 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:31.744777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:31.864999 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:31.864964    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:31.980008 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:31.979976    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:32.540497 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:32.540462    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:32.649966 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:32.649938    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:36.008436 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.008394    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:54:36.011330 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.009038    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="alertmanager" containerID="cri-o://9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea" gracePeriod=120
Nov 03 07:54:36.011330 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.009430    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy" containerID="cri-o://76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123" gracePeriod=120
Nov 03 07:54:36.011330 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.009598    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="prom-label-proxy" containerID="cri-o://c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0" gracePeriod=120
Nov 03 07:54:36.011330 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.009705    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="config-reloader" containerID="cri-o://e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0" gracePeriod=120
Nov 03 07:54:36.011330 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.009691    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-metric" containerID="cri-o://6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb" gracePeriod=120
Nov 03 07:54:36.011330 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.009792    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-web" containerID="cri-o://bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a" gracePeriod=120
Nov 03 07:54:36.749132 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749095    6430 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0" exitCode=0
Nov 03 07:54:36.749132 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749119    6430 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb" exitCode=0
Nov 03 07:54:36.749132 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749125    6430 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123" exitCode=0
Nov 03 07:54:36.749132 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749130    6430 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0" exitCode=0
Nov 03 07:54:36.749132 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749135    6430 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea" exitCode=0
Nov 03 07:54:36.749414 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749176    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0"}
Nov 03 07:54:36.749414 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749245    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb"}
Nov 03 07:54:36.749414 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749272    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123"}
Nov 03 07:54:36.749414 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749297    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0"}
Nov 03 07:54:36.749414 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:36.749319    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea"}
Nov 03 07:54:37.259469 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.259446    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.358448 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358370    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358448 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358411    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358448 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358435    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358732 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358510    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358732 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358558    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358732 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358601    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358732 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358638    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358732 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358678    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.358732 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358723    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.359129 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358771    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.359129 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358814    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.359129 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358845    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.359129 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.358900    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\") pod \"84caeca2-05a1-4588-b70e-c85607c1d219\" (UID: \"84caeca2-05a1-4588-b70e-c85607c1d219\") "
Nov 03 07:54:37.359129 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.359009    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:37.359488 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.359310    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-metrics-client-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.359488 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.359417    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:37.360356 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.360321    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:54:37.362602 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.362574    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:37.362751 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.362726    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:37.362907 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.362884    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:37.362907 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.362588    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "kube-api-access-s2xn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:37.363096 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.363016    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:37.363170 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.363122    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:37.363340 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.363311    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:54:37.363438 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.363417    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:37.366227 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.366201    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:37.372034 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.372012    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config" pod "84caeca2-05a1-4588-b70e-c85607c1d219" (UID: "84caeca2-05a1-4588-b70e-c85607c1d219"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:37.460417 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460389    6430 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-main-db\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460417 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460415    6430 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-web-config\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460430    6430 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460444    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460480    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xn2\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-kube-api-access-s2xn2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460497    6430 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-cluster-tls-config\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460511    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460525    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460541    6430 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84caeca2-05a1-4588-b70e-c85607c1d219-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460555    6430 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84caeca2-05a1-4588-b70e-c85607c1d219-tls-assets\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460569    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84caeca2-05a1-4588-b70e-c85607c1d219-secret-alertmanager-main-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.460603 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.460585    6430 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84caeca2-05a1-4588-b70e-c85607c1d219-config-out\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:37.756104 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.756054    6430 generic.go:356] "Generic (PLEG): container finished" podID="84caeca2-05a1-4588-b70e-c85607c1d219" containerID="bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a" exitCode=0
Nov 03 07:54:37.756104 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.756094    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a"}
Nov 03 07:54:37.756301 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.756137    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.756301 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.756172    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84caeca2-05a1-4588-b70e-c85607c1d219","Type":"ContainerDied","Data":"e28cd8256a9ac99a24dac1f744a1684a0704c0a79ee6974b61cada7dfd5485a9"}
Nov 03 07:54:37.756301 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.756212    6430 scope.go:122] "RemoveContainer" containerID="c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0"
Nov 03 07:54:37.764176 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.764156    6430 scope.go:122] "RemoveContainer" containerID="6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb"
Nov 03 07:54:37.771022 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.771002    6430 scope.go:122] "RemoveContainer" containerID="76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123"
Nov 03 07:54:37.777491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.777472    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:54:37.778200 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.778184    6430 scope.go:122] "RemoveContainer" containerID="bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a"
Nov 03 07:54:37.783479 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.783442    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:54:37.786079 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.786047    6430 scope.go:122] "RemoveContainer" containerID="e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0"
Nov 03 07:54:37.792865 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.792849    6430 scope.go:122] "RemoveContainer" containerID="9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea"
Nov 03 07:54:37.799488 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.799469    6430 scope.go:122] "RemoveContainer" containerID="2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700"
Nov 03 07:54:37.806389 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.806372    6430 scope.go:122] "RemoveContainer" containerID="c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0"
Nov 03 07:54:37.806968 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:37.806855    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0\": container with ID starting with c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0 not found: ID does not exist" containerID="c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0"
Nov 03 07:54:37.808641 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.807147    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0"} err="failed to get container status \"c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0\": rpc error: code = NotFound desc = could not find container \"c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0\": container with ID starting with c974add50fbccf3455f760de82c225d0d92a8c06fac0fd24367acbd0c881dbb0 not found: ID does not exist"
Nov 03 07:54:37.808641 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.807230    6430 scope.go:122] "RemoveContainer" containerID="6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809264    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809773    6430 cpu_manager.go:414] "containerMap: removing container" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="init-config-reloader"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809791    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="init-config-reloader"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809809    6430 cpu_manager.go:414] "containerMap: removing container" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809817    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809847    6430 cpu_manager.go:414] "containerMap: removing container" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-web"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809858    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-web"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809875    6430 cpu_manager.go:414] "containerMap: removing container" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-metric"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809882    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-metric"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809896    6430 cpu_manager.go:414] "containerMap: removing container" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="config-reloader"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809903    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="config-reloader"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809912    6430 cpu_manager.go:414] "containerMap: removing container" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="alertmanager"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809919    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="alertmanager"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809935    6430 cpu_manager.go:414] "containerMap: removing container" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="prom-label-proxy"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.809942    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="prom-label-proxy"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810036    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810056    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-web"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810086    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="kube-rbac-proxy-metric"
Nov 03 07:54:37.810087 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810095    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="config-reloader"
Nov 03 07:54:37.810913 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810110    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="alertmanager"
Nov 03 07:54:37.810913 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810119    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="prom-label-proxy"
Nov 03 07:54:37.810913 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810129    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" containerName="init-config-reloader"
Nov 03 07:54:37.811011 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:37.810970    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb\": container with ID starting with 6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb not found: ID does not exist" containerID="6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb"
Nov 03 07:54:37.811011 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.810997    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb"} err="failed to get container status \"6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb\": rpc error: code = NotFound desc = could not find container \"6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb\": container with ID starting with 6600efbd96e93a0e401c423b9e17914196e94caa0dea878cdc403a2daeb32bdb not found: ID does not exist"
Nov 03 07:54:37.811132 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.811020    6430 scope.go:122] "RemoveContainer" containerID="76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123"
Nov 03 07:54:37.811485 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:37.811446    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123\": container with ID starting with 76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123 not found: ID does not exist" containerID="76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123"
Nov 03 07:54:37.811807 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.811779    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123"} err="failed to get container status \"76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123\": rpc error: code = NotFound desc = could not find container \"76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123\": container with ID starting with 76f7516ef53308ee7072ede38060db4a1631f26f0624effb39dad47ec7cfc123 not found: ID does not exist"
Nov 03 07:54:37.811858 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.811810    6430 scope.go:122] "RemoveContainer" containerID="bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a"
Nov 03 07:54:37.812121 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:37.812093    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a\": container with ID starting with bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a not found: ID does not exist" containerID="bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a"
Nov 03 07:54:37.812196 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.812131    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a"} err="failed to get container status \"bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a\": rpc error: code = NotFound desc = could not find container \"bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a\": container with ID starting with bc1d9a19084996c9fe5d4bdde8a84d0324c7d1196083739136fef04496bb5f0a not found: ID does not exist"
Nov 03 07:54:37.812196 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.812153    6430 scope.go:122] "RemoveContainer" containerID="e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0"
Nov 03 07:54:37.812468 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:37.812444    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0\": container with ID starting with e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0 not found: ID does not exist" containerID="e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0"
Nov 03 07:54:37.812559 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.812472    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0"} err="failed to get container status \"e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0\": rpc error: code = NotFound desc = could not find container \"e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0\": container with ID starting with e984b9dadd8930efa810301cb2939a18cbfe418e515bc52d19a9ca068396f6a0 not found: ID does not exist"
Nov 03 07:54:37.812559 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.812487    6430 scope.go:122] "RemoveContainer" containerID="9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea"
Nov 03 07:54:37.812735 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:37.812718    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea\": container with ID starting with 9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea not found: ID does not exist" containerID="9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea"
Nov 03 07:54:37.812825 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.812740    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea"} err="failed to get container status \"9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea\": rpc error: code = NotFound desc = could not find container \"9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea\": container with ID starting with 9e87c6af9cfb9a156943f0c80a8bb46deeca54b33535fd65196252d0b09482ea not found: ID does not exist"
Nov 03 07:54:37.812825 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.812753    6430 scope.go:122] "RemoveContainer" containerID="2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700"
Nov 03 07:54:37.812991 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:37.812978    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700\": container with ID starting with 2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700 not found: ID does not exist" containerID="2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700"
Nov 03 07:54:37.813033 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.812995    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700"} err="failed to get container status \"2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700\": rpc error: code = NotFound desc = could not find container \"2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700\": container with ID starting with 2f3e9ead8325ea59b0dc8466d9940d5e5a126a197ce7e208f908430e1936a700 not found: ID does not exist"
Nov 03 07:54:37.814466 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.814449    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.817639 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.817594    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 07:54:37.817639 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.817637    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 07:54:37.817800 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.817675    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 07:54:37.817904 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.817888    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 07:54:37.818047 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.818009    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 07:54:37.818047 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.818036    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 07:54:37.818212 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.818037    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 07:54:37.818212 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.818119    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c4vhz\""
Nov 03 07:54:37.818212 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.818123    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 07:54:37.818956 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.818940    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 07:54:37.826443 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.826420    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:54:37.963958 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.963918    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.963958 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.963966    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/679d7319-fe60-433b-b49f-42862dedd70b-config-out\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964208 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.963982    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964208 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964052    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964208 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/679d7319-fe60-433b-b49f-42862dedd70b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964208 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964169    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679d7319-fe60-433b-b49f-42862dedd70b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964208 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964190    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clr82\" (UniqueName: \"kubernetes.io/projected/679d7319-fe60-433b-b49f-42862dedd70b-kube-api-access-clr82\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964357 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964226    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-web-config\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964357 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/679d7319-fe60-433b-b49f-42862dedd70b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964420 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964358    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964420 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964392    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/679d7319-fe60-433b-b49f-42862dedd70b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964480 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964428    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:37.964480 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:37.964448    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-config-volume\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065589 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065525    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/679d7319-fe60-433b-b49f-42862dedd70b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065589 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065556    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065589 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065572    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/679d7319-fe60-433b-b49f-42862dedd70b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065589 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065588    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065817 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065746    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-config-volume\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065817 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065808    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065927 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065857    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/679d7319-fe60-433b-b49f-42862dedd70b-config-out\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065927 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065886    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.065927 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065914    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.066090 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065944    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/679d7319-fe60-433b-b49f-42862dedd70b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.066090 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.065976    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679d7319-fe60-433b-b49f-42862dedd70b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.066090 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.066007    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clr82\" (UniqueName: \"kubernetes.io/projected/679d7319-fe60-433b-b49f-42862dedd70b-kube-api-access-clr82\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.066090 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.066026    6430 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 07:54:38.066090 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.066054    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-web-config\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.066339 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.066228    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/679d7319-fe60-433b-b49f-42862dedd70b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.066339 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.066331    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/679d7319-fe60-433b-b49f-42862dedd70b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.067050 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.067021    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679d7319-fe60-433b-b49f-42862dedd70b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069227 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069202    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-web-config\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069328 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069205    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-config-volume\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069328 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069244    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069328 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069267    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/679d7319-fe60-433b-b49f-42862dedd70b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069328 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069295    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069553 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069340    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069614 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069602    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.069992 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.069972    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/679d7319-fe60-433b-b49f-42862dedd70b-config-out\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.070983 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.070961    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/679d7319-fe60-433b-b49f-42862dedd70b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.073277 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.073258    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clr82\" (UniqueName: \"kubernetes.io/projected/679d7319-fe60-433b-b49f-42862dedd70b-kube-api-access-clr82\") pod \"alertmanager-main-0\" (UID: \"679d7319-fe60-433b-b49f-42862dedd70b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.130897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.130867    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 07:54:38.410093 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.409988    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 07:54:38.417520 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:54:38.417491    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679d7319_fe60_433b_b49f_42862dedd70b.slice/crio-91e8631e139d920ae8a2fbe011ae8beead444125acdfdbd6da1532642da1e01d WatchSource:0}: Error finding container 91e8631e139d920ae8a2fbe011ae8beead444125acdfdbd6da1532642da1e01d: Status 404 returned error can't find the container with id 91e8631e139d920ae8a2fbe011ae8beead444125acdfdbd6da1532642da1e01d
Nov 03 07:54:38.614105 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.614050    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84caeca2-05a1-4588-b70e-c85607c1d219" path="/var/lib/kubelet/pods/84caeca2-05a1-4588-b70e-c85607c1d219/volumes"
Nov 03 07:54:38.762273 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.762238    6430 generic.go:356] "Generic (PLEG): container finished" podID="679d7319-fe60-433b-b49f-42862dedd70b" containerID="732a226de5c72f9c2eded6d60ec037d2febe4a86c154d9a34edb1cb966177728" exitCode=0
Nov 03 07:54:38.762466 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.762318    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerDied","Data":"732a226de5c72f9c2eded6d60ec037d2febe4a86c154d9a34edb1cb966177728"}
Nov 03 07:54:38.762466 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.762392    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerStarted","Data":"91e8631e139d920ae8a2fbe011ae8beead444125acdfdbd6da1532642da1e01d"}
Nov 03 07:54:38.998184 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:38.998159    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-254xf"
Nov 03 07:54:39.770770 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:39.770733    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerStarted","Data":"2d54b68e4dd523522fece3c928db2b0bb8c6a92e21b1cec91b91d64c07005199"}
Nov 03 07:54:39.771147 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:39.770850    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerStarted","Data":"db3ebd0a12207c216a1a67558e53490c4146f37ced61ab872e3d607aa16d636d"}
Nov 03 07:54:39.771147 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:39.770893    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerStarted","Data":"c66f3c97a2ed61f2e014b5551efa17f7d722c622b720f1828c3d272d0ca07623"}
Nov 03 07:54:39.771147 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:39.770930    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerStarted","Data":"5f4282157193fbce09fdc09ee0a99dbfbfd8ee0a9a048cde463c8fbc40be7027"}
Nov 03 07:54:39.771147 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:39.770967    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerStarted","Data":"7f6bbb6c17fef50d2a9eaa1e4467b6ebc2b627b0f271020848539e1d530d2d28"}
Nov 03 07:54:39.771147 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:39.771000    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"679d7319-fe60-433b-b49f-42862dedd70b","Type":"ContainerStarted","Data":"4b9cc990a1f69ee94b0a11d2b92e6cca7b34c9743cd107e2990afeaa728d92bd"}
Nov 03 07:54:39.793635 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:39.793583    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.781420367 podStartE2EDuration="2.793566744s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.146377ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:39.792936709 +0000 UTC m=+29.295772234" watchObservedRunningTime="2026-11-03 07:54:39.793566744 +0000 UTC m=+29.296402258"
Nov 03 07:54:41.899853 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:41.899818    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:42.011103 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:42.011048    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:43.320193 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:43.320156    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:43.423082 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:43.423028    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:45.178713 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:45.178682    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k5mr4"
Nov 03 07:54:56.893241 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:56.893208    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:57.001123 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:57.001090    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:57.110383 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:54:57.110352    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:58.373155 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:54:58.373115    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:55:05.003953 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:55:05.003919    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:55:05.107279 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:55:05.107249    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:55:10.968713 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:55:10.968680    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:55:26.867813 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:55:26.867750    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:55:26.977028 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:55:26.976995    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:55:28.374253 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:55:28.374219    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:55:51.612299 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:55:51.612224    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:55:58.373202 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:55:58.373165    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:56:11.016051 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:56:11.015999    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:56:14.818509 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:56:14.818474    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:56:28.374107 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:56:28.374052    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:56:33.022704 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:56:33.022654    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:56:33.127612 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:56:33.127580    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:56:58.373660 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:56:58.373624    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:57:11.065621 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:11.065583    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:13.225851 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:13.225820    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:13.378242 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:13.378192    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:13.515711 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:13.515638    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:14.024768 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.024711    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"]
Nov 03 07:57:14.028099 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.028077    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.034591 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.034570    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 07:57:14.034712 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.034629    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 07:57:14.034826 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.034808    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8mwhh\""
Nov 03 07:57:14.046458 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.046430    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"]
Nov 03 07:57:14.060154 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.060116    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.060284 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.060170    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nws69\" (UniqueName: \"kubernetes.io/projected/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-kube-api-access-nws69\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.060284 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.060264    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.161437 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.161399    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.161541 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.161452    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nws69\" (UniqueName: \"kubernetes.io/projected/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-kube-api-access-nws69\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.161541 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.161479    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.161835 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.161804    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.161869 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.161811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.176015 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.175987    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nws69\" (UniqueName: \"kubernetes.io/projected/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-kube-api-access-nws69\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.343137 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.343036    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:14.537214 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.537185    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"]
Nov 03 07:57:14.540562 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.540539    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.543936 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.543836    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 07:57:14.543936 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.543836    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 07:57:14.543936 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.543869    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 07:57:14.543936 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.543888    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 07:57:14.543936 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.543891    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 07:57:14.543936 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.543906    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 07:57:14.543936 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.543938    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vk5zp\""
Nov 03 07:57:14.553490 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.553469    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"]
Nov 03 07:57:14.587131 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.587106    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:14.621189 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.621120    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"]
Nov 03 07:57:14.628573 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:57:14.628291    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cabc20a_eb90_4f43_a6a5_1fdbf7604a97.slice/crio-0f85434c3eab0b5216ab42a54982bca0635af671a703b2ea3666c7c192666787 WatchSource:0}: Error finding container 0f85434c3eab0b5216ab42a54982bca0635af671a703b2ea3666c7c192666787: Status 404 returned error can't find the container with id 0f85434c3eab0b5216ab42a54982bca0635af671a703b2ea3666c7c192666787
Nov 03 07:57:14.630180 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.630162    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:57:14.665429 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.665405    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmkr\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-kube-api-access-kjmkr\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.665553 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.665469    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/acea5f02-57a7-42cf-8a6f-808d02af7049-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.665607 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.665559    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.665899 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.665871    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.666044 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.666027    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.666223 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.666205    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.666286 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.666253    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.702493 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.702432    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:14.767577 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.767553    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.767657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.767584    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.767657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.767612    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.767752 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.767662    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmkr\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-kube-api-access-kjmkr\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.767752 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.767697    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/acea5f02-57a7-42cf-8a6f-808d02af7049-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.767752 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.767719    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.767752 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.767747    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.768573 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.768478    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.770151 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.770129    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/acea5f02-57a7-42cf-8a6f-808d02af7049-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.770249 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.770170    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.770249 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.770175    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.770355 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.770252    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.774852 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.774825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.775449 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.775431    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmkr\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-kube-api-access-kjmkr\") pod \"istiod-openshift-gateway-7876bfcb74-4jgnp\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:14.852515 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:14.852494    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:15.130589 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:15.130550    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"]
Nov 03 07:57:15.134760 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:57:15.134735    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacea5f02_57a7_42cf_8a6f_808d02af7049.slice/crio-d81f02562530f9ee7e5e6315ef92c2006f0744deef7f8744898d15804c2cf3d8 WatchSource:0}: Error finding container d81f02562530f9ee7e5e6315ef92c2006f0744deef7f8744898d15804c2cf3d8: Status 404 returned error can't find the container with id d81f02562530f9ee7e5e6315ef92c2006f0744deef7f8744898d15804c2cf3d8
Nov 03 07:57:15.253948 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:15.253916    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp" event={"ID":"acea5f02-57a7-42cf-8a6f-808d02af7049","Type":"ContainerStarted","Data":"d81f02562530f9ee7e5e6315ef92c2006f0744deef7f8744898d15804c2cf3d8"}
Nov 03 07:57:15.255167 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:15.255147    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g" event={"ID":"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97","Type":"ContainerStarted","Data":"0f85434c3eab0b5216ab42a54982bca0635af671a703b2ea3666c7c192666787"}
Nov 03 07:57:15.586244 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:15.586161    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:20.131968 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.131939    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 07:57:20.132263 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.132001    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 07:57:20.276491 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.276452    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp" event={"ID":"acea5f02-57a7-42cf-8a6f-808d02af7049","Type":"ContainerStarted","Data":"028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a"}
Nov 03 07:57:20.276745 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.276721    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:20.296156 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.296093    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp" podStartSLOduration=1.301729322 podStartE2EDuration="6.296051131s" totalImagesPullingTime="4.994321809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:20.29589486 +0000 UTC m=+189.798730385" watchObservedRunningTime="2026-11-03 07:57:20.296051131 +0000 UTC m=+189.798886646"
Nov 03 07:57:20.613289 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.613200    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:20.715623 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.715591    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:20.962626 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:20.962584    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:21.070744 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:21.070716    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:21.267512 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:21.267418    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:21.282973 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:21.282943    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:57:23.572896 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:23.572865    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:24.293021 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:24.292977    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g" event={"ID":"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97","Type":"ContainerStarted","Data":"40eddcdae9c7cb1fc90ad6610b721b4b1d1d02d5c808e7413ec1edf34476f859"}
Nov 03 07:57:25.297454 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:25.297419    6430 generic.go:356] "Generic (PLEG): container finished" podID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerID="40eddcdae9c7cb1fc90ad6610b721b4b1d1d02d5c808e7413ec1edf34476f859" exitCode=0
Nov 03 07:57:25.297840 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:25.297470    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g" event={"ID":"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97","Type":"ContainerDied","Data":"40eddcdae9c7cb1fc90ad6610b721b4b1d1d02d5c808e7413ec1edf34476f859"}
Nov 03 07:57:28.308844 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:28.308758    6430 generic.go:356] "Generic (PLEG): container finished" podID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerID="023ebde9cdedc9ea1223ece98e0f87346eb655cce8fcd16c995a27ca6996954b" exitCode=0
Nov 03 07:57:28.309267 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:28.308850    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g" event={"ID":"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97","Type":"ContainerDied","Data":"023ebde9cdedc9ea1223ece98e0f87346eb655cce8fcd16c995a27ca6996954b"}
Nov 03 07:57:28.373322 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:57:28.373285    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:57:37.342116 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:37.342033    6430 generic.go:356] "Generic (PLEG): container finished" podID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerID="c92bede2e91471e0eedad7e6600ea77e9c8e883e2db333d0fc79a3627728bc9a" exitCode=0
Nov 03 07:57:37.342448 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:37.342116    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g" event={"ID":"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97","Type":"ContainerDied","Data":"c92bede2e91471e0eedad7e6600ea77e9c8e883e2db333d0fc79a3627728bc9a"}
Nov 03 07:57:38.486741 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.486717    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:38.564462 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.564429    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-kube-api-access-nws69\" (UniqueName: \"kubernetes.io/projected/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-kube-api-access-nws69\") pod \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") "
Nov 03 07:57:38.564625 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.564526    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-util\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-util\") pod \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") "
Nov 03 07:57:38.564625 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.564610    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-bundle\") pod \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\" (UID: \"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97\") "
Nov 03 07:57:38.565324 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.565294    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-bundle" pod "9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" (UID: "9cabc20a-eb90-4f43-a6a5-1fdbf7604a97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:38.566764 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.566743    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-kube-api-access-nws69" pod "9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" (UID: "9cabc20a-eb90-4f43-a6a5-1fdbf7604a97"). InnerVolumeSpecName "kube-api-access-nws69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:57:38.570513 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.570487    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-util" pod "9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" (UID: "9cabc20a-eb90-4f43-a6a5-1fdbf7604a97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:38.665123 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.665094    6430 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-bundle\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:38.665123 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.665122    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nws69\" (UniqueName: \"kubernetes.io/projected/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-kube-api-access-nws69\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:38.665252 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:38.665139    6430 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cabc20a-eb90-4f43-a6a5-1fdbf7604a97-util\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:39.351355 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:39.351326    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g"
Nov 03 07:57:39.351549 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:39.351323    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnp85g" event={"ID":"9cabc20a-eb90-4f43-a6a5-1fdbf7604a97","Type":"ContainerDied","Data":"0f85434c3eab0b5216ab42a54982bca0635af671a703b2ea3666c7c192666787"}
Nov 03 07:57:39.351549 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:39.351468    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f85434c3eab0b5216ab42a54982bca0635af671a703b2ea3666c7c192666787"
Nov 03 07:57:54.534095 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:54.534041    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:55.317989 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.317958    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"]
Nov 03 07:57:55.318262 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.318250    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerName="util"
Nov 03 07:57:55.318305 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.318263    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerName="util"
Nov 03 07:57:55.318305 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.318273    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerName="pull"
Nov 03 07:57:55.318305 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.318278    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerName="pull"
Nov 03 07:57:55.318305 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.318293    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerName="extract"
Nov 03 07:57:55.318305 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.318298    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerName="extract"
Nov 03 07:57:55.318449 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.318337    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cabc20a-eb90-4f43-a6a5-1fdbf7604a97" containerName="extract"
Nov 03 07:57:55.321179 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.321163    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.351741 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.351719    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"]
Nov 03 07:57:55.403186 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.403151    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 07:57:55.403381 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.403332    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 07:57:55.403381 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.403375    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 07:57:55.403517 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.403479    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 07:57:55.403572 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.403529    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cclgr\""
Nov 03 07:57:55.403613 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.403598    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 07:57:55.489610 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.489578    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.489770 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.489627    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.489770 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.489690    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vt8n\" (UniqueName: \"kubernetes.io/projected/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-kube-api-access-4vt8n\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.489770 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.489724    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.489928 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.489827    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.489928 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.489874    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.590875 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.590789    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.590875 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.590833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.590875 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.590854    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vt8n\" (UniqueName: \"kubernetes.io/projected/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-kube-api-access-4vt8n\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.591404 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.590988    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.591404 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.591097    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.591404 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.591129    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.591561 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.591481    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.593533 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.593503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.593661 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.593565    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.593730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.593714    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.593937 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.593919    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.598974 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.598953    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vt8n\" (UniqueName: \"kubernetes.io/projected/19c36ccd-5593-4a5d-9c5b-4d60c58dab44-kube-api-access-4vt8n\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-xwwvj\" (UID: \"19c36ccd-5593-4a5d-9c5b-4d60c58dab44\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.634865 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.634829    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:57:55.746666 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.746623    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:55.928150 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:55.928020    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"]
Nov 03 07:57:55.934365 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:57:55.934337    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c36ccd_5593_4a5d_9c5b_4d60c58dab44.slice/crio-e89af3681ac241bd7bf126f9b424366f61dea1ba68fef0689bd448e968810e07 WatchSource:0}: Error finding container e89af3681ac241bd7bf126f9b424366f61dea1ba68fef0689bd448e968810e07: Status 404 returned error can't find the container with id e89af3681ac241bd7bf126f9b424366f61dea1ba68fef0689bd448e968810e07
Nov 03 07:57:56.407164 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:56.407133    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj" event={"ID":"19c36ccd-5593-4a5d-9c5b-4d60c58dab44","Type":"ContainerStarted","Data":"e89af3681ac241bd7bf126f9b424366f61dea1ba68fef0689bd448e968810e07"}
Nov 03 07:57:56.537050 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:56.537022    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:57.934776 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:57.934739    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:58.089610 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:58.089574    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:57:58.378248 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:57:58.378154    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:57:59.419739 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:59.419661    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj" event={"ID":"19c36ccd-5593-4a5d-9c5b-4d60c58dab44","Type":"ContainerStarted","Data":"89682326f28ae2aa7d55cfc90f96de98567c05ed3bdf53b77e0e465b1c61177f"}
Nov 03 07:57:59.940466 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:57:59.940437    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:00.425102 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:00.424995    6430 generic.go:356] "Generic (PLEG): container finished" podID="19c36ccd-5593-4a5d-9c5b-4d60c58dab44" containerID="89682326f28ae2aa7d55cfc90f96de98567c05ed3bdf53b77e0e465b1c61177f" exitCode=0
Nov 03 07:58:00.425102 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:00.425042    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj" event={"ID":"19c36ccd-5593-4a5d-9c5b-4d60c58dab44","Type":"ContainerDied","Data":"89682326f28ae2aa7d55cfc90f96de98567c05ed3bdf53b77e0e465b1c61177f"}
Nov 03 07:58:01.430337 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:01.430304    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj" event={"ID":"19c36ccd-5593-4a5d-9c5b-4d60c58dab44","Type":"ContainerStarted","Data":"d38d801b7696156f8d20c1d64535680cf4836ecca8a434b8e6bd89b890921a20"}
Nov 03 07:58:01.430771 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:01.430434    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:58:01.460732 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:01.460691    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj" podStartSLOduration=2.560057928 podStartE2EDuration="6.460676848s" totalImagesPullingTime="3.490174353s" totalInitContainerRuntime="410.444567ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:01.458870883 +0000 UTC m=+230.961706396" watchObservedRunningTime="2026-11-03 07:58:01.460676848 +0000 UTC m=+230.963512431"
Nov 03 07:58:03.340663 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:03.340633    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:03.467333 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:03.467297    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:05.820509 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:05.820478    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:06.391842 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:06.391813    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:06.511332 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:06.511305    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:11.117289 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:11.117252    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:11.950413 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:11.950378    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:12.099307 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:12.099272    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:12.435683 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:12.435659    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-xwwvj"
Nov 03 07:58:13.801195 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:13.801157    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:13.949294 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:13.949258    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:14.414397 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:14.414361    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:17.519235 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:17.519201    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:21.938976 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:21.938904    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:22.525042 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:22.525009    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:22.750719 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:22.750691    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:22.857845 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:22.857765    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:23.192269 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:23.192239    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:23.393466 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:23.393436    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:25.976991 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:25.976959    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:27.387443 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:27.387415    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:27.514912 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:27.514885    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:27.692604 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:27.692578    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:28.127123 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:28.127003    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:28.373553 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:58:28.373519    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:58:29.164573 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:29.164535    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:30.738707 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:30.738675    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:30.870709 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:30.870679    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:31.546749 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.546720    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:31.769424 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.769388    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:31.884686 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.884602    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"]
Nov 03 07:58:31.890043 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.890021    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:31.904848 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.904819    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"]
Nov 03 07:58:31.943185 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.943151    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 07:58:31.943454 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.943437    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gzpwd\""
Nov 03 07:58:31.943454 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.943443    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 07:58:31.949841 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.949819    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23318d3-5990-4aea-9c49-e821bb20fd56-tmp\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:31.949997 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.949848    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23318d3-5990-4aea-9c49-e821bb20fd56-tls-certs\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:31.949997 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.949882    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4sl\" (UniqueName: \"kubernetes.io/projected/e23318d3-5990-4aea-9c49-e821bb20fd56-kube-api-access-8x4sl\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:31.953055 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:31.953030    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:32.051010 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.050981    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23318d3-5990-4aea-9c49-e821bb20fd56-tmp\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:32.051010 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.051014    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23318d3-5990-4aea-9c49-e821bb20fd56-tls-certs\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:32.051241 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.051038    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4sl\" (UniqueName: \"kubernetes.io/projected/e23318d3-5990-4aea-9c49-e821bb20fd56-kube-api-access-8x4sl\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:32.053389 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.053362    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23318d3-5990-4aea-9c49-e821bb20fd56-tmp\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:32.053626 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.053610    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23318d3-5990-4aea-9c49-e821bb20fd56-tls-certs\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:32.061132 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.061110    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4sl\" (UniqueName: \"kubernetes.io/projected/e23318d3-5990-4aea-9c49-e821bb20fd56-kube-api-access-8x4sl\") pod \"kube-auth-proxy-6cf769c4c7-2hhrl\" (UID: \"e23318d3-5990-4aea-9c49-e821bb20fd56\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:32.115139 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.115051    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:32.201358 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.201331    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"
Nov 03 07:58:32.262585 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.262420    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:32.412291 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.412256    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:32.471619 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.471553    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl"]
Nov 03 07:58:32.475650 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:58:32.475621    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23318d3_5990_4aea_9c49_e821bb20fd56.slice/crio-fa7e2984eb0fbdbdd00b299afa151e00947115c8d7368a65a956a64f85e6ff27 WatchSource:0}: Error finding container fa7e2984eb0fbdbdd00b299afa151e00947115c8d7368a65a956a64f85e6ff27: Status 404 returned error can't find the container with id fa7e2984eb0fbdbdd00b299afa151e00947115c8d7368a65a956a64f85e6ff27
Nov 03 07:58:32.541757 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.541729    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl" event={"ID":"e23318d3-5990-4aea-9c49-e821bb20fd56","Type":"ContainerStarted","Data":"fa7e2984eb0fbdbdd00b299afa151e00947115c8d7368a65a956a64f85e6ff27"}
Nov 03 07:58:32.705145 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.705114    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"]
Nov 03 07:58:32.707797 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.707777    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"
Nov 03 07:58:32.719844 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.719826    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 07:58:32.719955 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.719916    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 07:58:32.720007 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.719975    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 07:58:32.723538 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.723482    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"]
Nov 03 07:58:32.758675 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.758652    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6dtb\" (UniqueName: \"kubernetes.io/projected/145aee33-7914-4590-bc10-6c79746d247e-kube-api-access-d6dtb\") pod \"dns-operator-controller-manager-8557cbdb84-q2k2f\" (UID: \"145aee33-7914-4590-bc10-6c79746d247e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"
Nov 03 07:58:32.769535 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.769514    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n8mq8\""
Nov 03 07:58:32.859657 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.859632    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6dtb\" (UniqueName: \"kubernetes.io/projected/145aee33-7914-4590-bc10-6c79746d247e-kube-api-access-d6dtb\") pod \"dns-operator-controller-manager-8557cbdb84-q2k2f\" (UID: \"145aee33-7914-4590-bc10-6c79746d247e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"
Nov 03 07:58:32.880998 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:32.880972    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6dtb\" (UniqueName: \"kubernetes.io/projected/145aee33-7914-4590-bc10-6c79746d247e-kube-api-access-d6dtb\") pod \"dns-operator-controller-manager-8557cbdb84-q2k2f\" (UID: \"145aee33-7914-4590-bc10-6c79746d247e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"
Nov 03 07:58:33.019639 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:33.019559    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"
Nov 03 07:58:33.293355 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:33.293253    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"]
Nov 03 07:58:33.298180 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:58:33.298155    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145aee33_7914_4590_bc10_6c79746d247e.slice/crio-618d8d942f6fb4f0095a2fb76705962b988777f1bb7c93e1599f958ba3e7ad4a WatchSource:0}: Error finding container 618d8d942f6fb4f0095a2fb76705962b988777f1bb7c93e1599f958ba3e7ad4a: Status 404 returned error can't find the container with id 618d8d942f6fb4f0095a2fb76705962b988777f1bb7c93e1599f958ba3e7ad4a
Nov 03 07:58:33.547976 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:33.547874    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f" event={"ID":"145aee33-7914-4590-bc10-6c79746d247e","Type":"ContainerStarted","Data":"618d8d942f6fb4f0095a2fb76705962b988777f1bb7c93e1599f958ba3e7ad4a"}
Nov 03 07:58:33.554575 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:33.554545    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:33.942955 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:33.942925    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:34.129326 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:34.129298    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:34.244454 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:34.244383    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:35.606596 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.606562    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"]
Nov 03 07:58:35.610400 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.610379    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.619351 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.618989    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 07:58:35.619351 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.619170    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-qxw79\""
Nov 03 07:58:35.619351 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.619320    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 07:58:35.624086 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.623153    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 07:58:35.626735 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.626373    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"]
Nov 03 07:58:35.685787 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.685754    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxjnk\" (UniqueName: \"kubernetes.io/projected/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-kube-api-access-fxjnk\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.685944 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.685876    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.685944 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.685923    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.685944 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.685940    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.778291 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.778213    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:35.786780 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.786751    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.786934 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.786803    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.786934 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.786830    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.786934 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.786914    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxjnk\" (UniqueName: \"kubernetes.io/projected/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-kube-api-access-fxjnk\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.787348 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.787323    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.787475 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.787454    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.787860 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.787815    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.795149 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.795124    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxjnk\" (UniqueName: \"kubernetes.io/projected/5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1-kube-api-access-fxjnk\") pod \"kserve-module-controller-manager-7566cf5fc8-p6rtl\" (UID: \"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:35.893436 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.893323    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:35.934711 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:35.934682    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:36.226959 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:36.226926    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"]
Nov 03 07:58:36.230816 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:58:36.230782    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc72eb0_6403_491f_bb27_dd3ccc6a2cc1.slice/crio-f49cd858298be83257e459d565f091ba266cfc8dd84e93339b4cb0fb107b28a5 WatchSource:0}: Error finding container f49cd858298be83257e459d565f091ba266cfc8dd84e93339b4cb0fb107b28a5: Status 404 returned error can't find the container with id f49cd858298be83257e459d565f091ba266cfc8dd84e93339b4cb0fb107b28a5
Nov 03 07:58:36.563604 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:36.563516    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl" event={"ID":"e23318d3-5990-4aea-9c49-e821bb20fd56","Type":"ContainerStarted","Data":"35fad0a0fe59f9363bd50021821124c0b6315c2f100e58fe175c763bd6c58bb1"}
Nov 03 07:58:36.564852 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:36.564823    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl" event={"ID":"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1","Type":"ContainerStarted","Data":"f49cd858298be83257e459d565f091ba266cfc8dd84e93339b4cb0fb107b28a5"}
Nov 03 07:58:36.583881 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:36.583828    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-2hhrl" podStartSLOduration=2.159816218 podStartE2EDuration="5.583810823s" totalImagesPullingTime="3.423994605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:36.583491955 +0000 UTC m=+266.086327470" watchObservedRunningTime="2026-11-03 07:58:36.583810823 +0000 UTC m=+266.086646337"
Nov 03 07:58:37.570735 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:37.570699    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f" event={"ID":"145aee33-7914-4590-bc10-6c79746d247e","Type":"ContainerStarted","Data":"a6e9dacc4c50264e8034a1f274dfc8314091f8ce5f6274ee1f77408b808a7d23"}
Nov 03 07:58:37.586762 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:37.586569    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f" podStartSLOduration=2.082216717 podStartE2EDuration="5.58655353s" totalImagesPullingTime="3.504336813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:37.585382513 +0000 UTC m=+267.088218027" watchObservedRunningTime="2026-11-03 07:58:37.58655353 +0000 UTC m=+267.089389039"
Nov 03 07:58:38.576495 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:38.576406    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl" event={"ID":"5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1","Type":"ContainerStarted","Data":"4519a2f0a26d131729a3e4cdff0ff2b8558a02dcf86be1048a1718cd230e3cbb"}
Nov 03 07:58:38.576865 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:38.576497    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"
Nov 03 07:58:38.594867 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:38.594828    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl" podStartSLOduration=1.5638200850000001 podStartE2EDuration="3.594816444s" totalImagesPullingTime="2.030996359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:38.594229128 +0000 UTC m=+268.097064642" watchObservedRunningTime="2026-11-03 07:58:38.594816444 +0000 UTC m=+268.097651957"
Nov 03 07:58:38.729040 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:38.729015    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:38.844878 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:38.844803    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:38.946950 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:38.946898    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:39.580484 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:39.580458    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:58:39.721801 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:39.721768    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:39.839091 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:39.838984    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:39.953195 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:39.953166    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:46.995263 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:46.995227    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:47.117441 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:47.117408    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:49.407217 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:49.407130    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:49.519355 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:49.519300    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:49.582526 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:49.582496    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-q2k2f"
Nov 03 07:58:53.223080 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.223027    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"]
Nov 03 07:58:53.226730 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.226710    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.229554 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.229527    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 07:58:53.229878 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.229862    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qv8vc\""
Nov 03 07:58:53.234860 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.234831    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"]
Nov 03 07:58:53.276209 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.276175    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:53.328020 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.327988    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-config-file\") pod \"limitador-limitador-74dfcfb894-bj2pt\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.328191 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.328050    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzft\" (UniqueName: \"kubernetes.io/projected/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-kube-api-access-tzzft\") pod \"limitador-limitador-74dfcfb894-bj2pt\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.334680 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.334654    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"]
Nov 03 07:58:53.410886 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.410850    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:53.429013 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.428977    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzft\" (UniqueName: \"kubernetes.io/projected/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-kube-api-access-tzzft\") pod \"limitador-limitador-74dfcfb894-bj2pt\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.429224 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.429100    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-config-file\") pod \"limitador-limitador-74dfcfb894-bj2pt\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.429729 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.429701    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-config-file\") pod \"limitador-limitador-74dfcfb894-bj2pt\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.440773 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.440747    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzft\" (UniqueName: \"kubernetes.io/projected/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-kube-api-access-tzzft\") pod \"limitador-limitador-74dfcfb894-bj2pt\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.540494 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.540421    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:53.566256 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.566222    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:53.815388 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.815293    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"]
Nov 03 07:58:53.819431 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:58:53.819402    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0797a0_6caa_4b4d_9c6a_7f1e3d06355a.slice/crio-0e47581651c4c916535d0b82fef6c7c1b7595f752391932525480260ba30fcdb WatchSource:0}: Error finding container 0e47581651c4c916535d0b82fef6c7c1b7595f752391932525480260ba30fcdb: Status 404 returned error can't find the container with id 0e47581651c4c916535d0b82fef6c7c1b7595f752391932525480260ba30fcdb
Nov 03 07:58:53.863403 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.863377    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:53.987197 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:53.987172    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:54.119437 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.119366    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6dsrq"]
Nov 03 07:58:54.123958 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.123936    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6dsrq"
Nov 03 07:58:54.131533 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.131511    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6k6d7\""
Nov 03 07:58:54.132766 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.132744    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6dsrq"]
Nov 03 07:58:54.198438 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.198409    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:54.235821 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.235791    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxhb\" (UniqueName: \"kubernetes.io/projected/12c71d0c-b1e1-450c-8e20-534885b60056-kube-api-access-hdxhb\") pod \"authorino-57f7f8659f-6dsrq\" (UID: \"12c71d0c-b1e1-450c-8e20-534885b60056\") " pod="kuadrant-system/authorino-57f7f8659f-6dsrq"
Nov 03 07:58:54.314781 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.314753    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:54.337151 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.337129    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxhb\" (UniqueName: \"kubernetes.io/projected/12c71d0c-b1e1-450c-8e20-534885b60056-kube-api-access-hdxhb\") pod \"authorino-57f7f8659f-6dsrq\" (UID: \"12c71d0c-b1e1-450c-8e20-534885b60056\") " pod="kuadrant-system/authorino-57f7f8659f-6dsrq"
Nov 03 07:58:54.350777 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.350753    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxhb\" (UniqueName: \"kubernetes.io/projected/12c71d0c-b1e1-450c-8e20-534885b60056-kube-api-access-hdxhb\") pod \"authorino-57f7f8659f-6dsrq\" (UID: \"12c71d0c-b1e1-450c-8e20-534885b60056\") " pod="kuadrant-system/authorino-57f7f8659f-6dsrq"
Nov 03 07:58:54.445389 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.445369    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6dsrq"
Nov 03 07:58:54.638488 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.638445    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt" event={"ID":"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a","Type":"ContainerStarted","Data":"0e47581651c4c916535d0b82fef6c7c1b7595f752391932525480260ba30fcdb"}
Nov 03 07:58:54.724434 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:54.724354    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6dsrq"]
Nov 03 07:58:54.728492 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:58:54.728464    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c71d0c_b1e1_450c_8e20_534885b60056.slice/crio-fdfa9f105041bf1e76d01d4febd29581f75ac0d38eece7e75d9432f058b48632 WatchSource:0}: Error finding container fdfa9f105041bf1e76d01d4febd29581f75ac0d38eece7e75d9432f058b48632: Status 404 returned error can't find the container with id fdfa9f105041bf1e76d01d4febd29581f75ac0d38eece7e75d9432f058b48632
Nov 03 07:58:55.642807 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:55.642775    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6dsrq" event={"ID":"12c71d0c-b1e1-450c-8e20-534885b60056","Type":"ContainerStarted","Data":"fdfa9f105041bf1e76d01d4febd29581f75ac0d38eece7e75d9432f058b48632"}
Nov 03 07:58:57.564103 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:57.564040    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:57.667630 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:57.667598    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:57.775127 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:57.775094    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:58.373605 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:58:58.373569    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:58:58.656584 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.656551    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6dsrq" event={"ID":"12c71d0c-b1e1-450c-8e20-534885b60056","Type":"ContainerStarted","Data":"60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac"}
Nov 03 07:58:58.657980 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.657959    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt" event={"ID":"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a","Type":"ContainerStarted","Data":"2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf"}
Nov 03 07:58:58.658134 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.658119    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:58:58.669881 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.669793    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-6dsrq" podStartSLOduration=1.5043115070000002 podStartE2EDuration="4.669775242s" totalImagesPullingTime="3.165463735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:58.668807035 +0000 UTC m=+288.171642550" watchObservedRunningTime="2026-11-03 07:58:58.669775242 +0000 UTC m=+288.172610757"
Nov 03 07:58:58.683447 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.683392    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt" podStartSLOduration=1.367845486 podStartE2EDuration="5.683374816s" totalImagesPullingTime="4.31552933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:58.683014913 +0000 UTC m=+288.185850442" watchObservedRunningTime="2026-11-03 07:58:58.683374816 +0000 UTC m=+288.186210330"
Nov 03 07:58:58.763702 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.763656    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:58:58.815407 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.815256    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6dsrq"]
Nov 03 07:58:58.881190 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:58:58.881151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:00.585542 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:00.585517    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p6rtl"
Nov 03 07:59:00.666043 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:00.666005    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-6dsrq" podUID="12c71d0c-b1e1-450c-8e20-534885b60056" containerName="authorino" containerID="cri-o://60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac" gracePeriod=30
Nov 03 07:59:00.905421 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:00.905400    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6dsrq"
Nov 03 07:59:00.992108 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:00.992083    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c71d0c-b1e1-450c-8e20-534885b60056-kube-api-access-hdxhb\" (UniqueName: \"kubernetes.io/projected/12c71d0c-b1e1-450c-8e20-534885b60056-kube-api-access-hdxhb\") pod \"12c71d0c-b1e1-450c-8e20-534885b60056\" (UID: \"12c71d0c-b1e1-450c-8e20-534885b60056\") "
Nov 03 07:59:00.994238 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:00.994210    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c71d0c-b1e1-450c-8e20-534885b60056-kube-api-access-hdxhb" pod "12c71d0c-b1e1-450c-8e20-534885b60056" (UID: "12c71d0c-b1e1-450c-8e20-534885b60056"). InnerVolumeSpecName "kube-api-access-hdxhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:59:01.093608 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.093578    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxhb\" (UniqueName: \"kubernetes.io/projected/12c71d0c-b1e1-450c-8e20-534885b60056-kube-api-access-hdxhb\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:01.675466 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.675435    6430 generic.go:356] "Generic (PLEG): container finished" podID="12c71d0c-b1e1-450c-8e20-534885b60056" containerID="60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac" exitCode=0
Nov 03 07:59:01.675939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.675477    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6dsrq"
Nov 03 07:59:01.675939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.675525    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6dsrq" event={"ID":"12c71d0c-b1e1-450c-8e20-534885b60056","Type":"ContainerDied","Data":"60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac"}
Nov 03 07:59:01.675939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.675564    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6dsrq" event={"ID":"12c71d0c-b1e1-450c-8e20-534885b60056","Type":"ContainerDied","Data":"fdfa9f105041bf1e76d01d4febd29581f75ac0d38eece7e75d9432f058b48632"}
Nov 03 07:59:01.675939 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.675579    6430 scope.go:122] "RemoveContainer" containerID="60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac"
Nov 03 07:59:01.684426 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.684409    6430 scope.go:122] "RemoveContainer" containerID="60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac"
Nov 03 07:59:01.684689 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:59:01.684670    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac\": container with ID starting with 60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac not found: ID does not exist" containerID="60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac"
Nov 03 07:59:01.684733 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.684697    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac"} err="failed to get container status \"60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac\": rpc error: code = NotFound desc = could not find container \"60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac\": container with ID starting with 60e5c0e1b10e4ed1f5ea98c17fe9a579d9864a18cb9d5399e61dc9fe97ee0fac not found: ID does not exist"
Nov 03 07:59:01.693260 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.693237    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6dsrq"]
Nov 03 07:59:01.696371 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:01.696351    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6dsrq"]
Nov 03 07:59:02.614890 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:02.614859    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c71d0c-b1e1-450c-8e20-534885b60056" path="/var/lib/kubelet/pods/12c71d0c-b1e1-450c-8e20-534885b60056/volumes"
Nov 03 07:59:07.810296 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:07.810257    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"]
Nov 03 07:59:07.810766 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:07.810552    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt" podUID="3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" containerName="limitador" containerID="cri-o://2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf" gracePeriod=30
Nov 03 07:59:07.811386 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:07.811248    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:59:07.970707 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:07.970655    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:08.078356 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.078267    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:08.363056 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.363032    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:59:08.449303 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.449270    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-config-file\" (UniqueName: \"kubernetes.io/configmap/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-config-file\") pod \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") "
Nov 03 07:59:08.449490 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.449365    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-kube-api-access-tzzft\" (UniqueName: \"kubernetes.io/projected/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-kube-api-access-tzzft\") pod \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\" (UID: \"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a\") "
Nov 03 07:59:08.449629 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.449599    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-config-file" pod "3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" (UID: "3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:59:08.451650 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.451626    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-kube-api-access-tzzft" pod "3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" (UID: "3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a"). InnerVolumeSpecName "kube-api-access-tzzft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:59:08.550852 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.550816    6430 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-config-file\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:08.550852 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.550846    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzzft\" (UniqueName: \"kubernetes.io/projected/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a-kube-api-access-tzzft\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:08.703267 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.703231    6430 generic.go:356] "Generic (PLEG): container finished" podID="3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" containerID="2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf" exitCode=0
Nov 03 07:59:08.703435 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.703276    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt" event={"ID":"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a","Type":"ContainerDied","Data":"2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf"}
Nov 03 07:59:08.703435 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.703290    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"
Nov 03 07:59:08.703435 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.703327    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bj2pt" event={"ID":"3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a","Type":"ContainerDied","Data":"0e47581651c4c916535d0b82fef6c7c1b7595f752391932525480260ba30fcdb"}
Nov 03 07:59:08.703435 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.703357    6430 scope.go:122] "RemoveContainer" containerID="2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf"
Nov 03 07:59:08.711556 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.711539    6430 scope.go:122] "RemoveContainer" containerID="2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf"
Nov 03 07:59:08.711823 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:59:08.711793    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf\": container with ID starting with 2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf not found: ID does not exist" containerID="2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf"
Nov 03 07:59:08.711901 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.711822    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf"} err="failed to get container status \"2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf\": rpc error: code = NotFound desc = could not find container \"2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf\": container with ID starting with 2cd555af6b09ed6518b18c7ab206a08b34365d17c41b6f21167a0372ab63afaf not found: ID does not exist"
Nov 03 07:59:08.716988 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.716966    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"]
Nov 03 07:59:08.722313 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:08.722293    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bj2pt"]
Nov 03 07:59:10.409266 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:10.409229    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:10.513252 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:10.513203    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:10.616379 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:10.616353    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" path="/var/lib/kubelet/pods/3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a/volumes"
Nov 03 07:59:11.167123 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:11.167095    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:18.051269 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.051234    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"]
Nov 03 07:59:18.051897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.051705    6430 cpu_manager.go:414] "containerMap: removing container" podUID="12c71d0c-b1e1-450c-8e20-534885b60056" containerName="authorino"
Nov 03 07:59:18.051897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.051725    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c71d0c-b1e1-450c-8e20-534885b60056" containerName="authorino"
Nov 03 07:59:18.051897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.051745    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" containerName="limitador"
Nov 03 07:59:18.051897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.051753    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" containerName="limitador"
Nov 03 07:59:18.051897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.051833    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c71d0c-b1e1-450c-8e20-534885b60056" containerName="authorino"
Nov 03 07:59:18.051897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.051846    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0797a0-6caa-4b4d-9c6a-7f1e3d06355a" containerName="limitador"
Nov 03 07:59:18.057371 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.057351    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.062681 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.062655    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"]
Nov 03 07:59:18.064630 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.064607    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 07:59:18.064745 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.064661    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 07:59:18.065005 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.064982    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qv8vc\""
Nov 03 07:59:18.095156 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.095127    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:18.234405 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.234367    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c8d6356-3592-4952-846d-3b7c992b6a3c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.234580 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.234494    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c8d6356-3592-4952-846d-3b7c992b6a3c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.234580 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.234550    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc55m\" (UniqueName: \"kubernetes.io/projected/3c8d6356-3592-4952-846d-3b7c992b6a3c-kube-api-access-nc55m\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.242706 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.242675    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:18.335533 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.335443    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c8d6356-3592-4952-846d-3b7c992b6a3c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.335533 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.335488    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc55m\" (UniqueName: \"kubernetes.io/projected/3c8d6356-3592-4952-846d-3b7c992b6a3c-kube-api-access-nc55m\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.335920 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.335625    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c8d6356-3592-4952-846d-3b7c992b6a3c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.336202 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.336182    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c8d6356-3592-4952-846d-3b7c992b6a3c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.338187 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.338168    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c8d6356-3592-4952-846d-3b7c992b6a3c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.342361 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.342339    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc55m\" (UniqueName: \"kubernetes.io/projected/3c8d6356-3592-4952-846d-3b7c992b6a3c-kube-api-access-nc55m\") pod \"kuadrant-console-plugin-66c4fb575d-qp7tx\" (UID: \"3c8d6356-3592-4952-846d-3b7c992b6a3c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.375655 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.375633    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"
Nov 03 07:59:18.644475 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.644366    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx"]
Nov 03 07:59:18.648357 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:59:18.648330    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c8d6356_3592_4952_846d_3b7c992b6a3c.slice/crio-ab9ac8b9dc9adfc6ff669bbd8ca82fb4482c87a4cedb43f394715906f0486e1c WatchSource:0}: Error finding container ab9ac8b9dc9adfc6ff669bbd8ca82fb4482c87a4cedb43f394715906f0486e1c: Status 404 returned error can't find the container with id ab9ac8b9dc9adfc6ff669bbd8ca82fb4482c87a4cedb43f394715906f0486e1c
Nov 03 07:59:18.747687 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:18.747655    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx" event={"ID":"3c8d6356-3592-4952-846d-3b7c992b6a3c","Type":"ContainerStarted","Data":"ab9ac8b9dc9adfc6ff669bbd8ca82fb4482c87a4cedb43f394715906f0486e1c"}
Nov 03 07:59:19.748555 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:19.748526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:19.861796 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:19.861764    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:20.342262 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.342225    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"]
Nov 03 07:59:20.345472 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.345452    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.348340 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.348318    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 07:59:20.348578 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.348555    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 07:59:20.353999 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.353978    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"]
Nov 03 07:59:20.372710 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.372685    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 07:59:20.372846 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.372714    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9xqj7\""
Nov 03 07:59:20.447375 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.447344    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:20.452577 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.452552    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6445\" (UniqueName: \"kubernetes.io/projected/21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe-kube-api-access-d6445\") pod \"postgres-7cdbf6dc44-t88sq\" (UID: \"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.452687 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.452607    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe-data\") pod \"postgres-7cdbf6dc44-t88sq\" (UID: \"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.553694 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.553658    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe-data\") pod \"postgres-7cdbf6dc44-t88sq\" (UID: \"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.553883 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.553798    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6445\" (UniqueName: \"kubernetes.io/projected/21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe-kube-api-access-d6445\") pod \"postgres-7cdbf6dc44-t88sq\" (UID: \"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.553993 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.553975    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe-data\") pod \"postgres-7cdbf6dc44-t88sq\" (UID: \"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.562210 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.562184    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6445\" (UniqueName: \"kubernetes.io/projected/21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe-kube-api-access-d6445\") pod \"postgres-7cdbf6dc44-t88sq\" (UID: \"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.658442 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.658370    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:20.929650 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:20.929579    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"]
Nov 03 07:59:20.932558 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:59:20.932532    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b7eca9_6d1d_49be_a7e7_48d27ccfb8fe.slice/crio-febb138adafad831991900dfedc213f98637e8d57932b01e7453307a52b62a00 WatchSource:0}: Error finding container febb138adafad831991900dfedc213f98637e8d57932b01e7453307a52b62a00: Status 404 returned error can't find the container with id febb138adafad831991900dfedc213f98637e8d57932b01e7453307a52b62a00
Nov 03 07:59:21.761818 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:21.761784    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq" event={"ID":"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe","Type":"ContainerStarted","Data":"febb138adafad831991900dfedc213f98637e8d57932b01e7453307a52b62a00"}
Nov 03 07:59:27.488912 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.488876    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:27.556561 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.556525    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-v6gdq"]
Nov 03 07:59:27.578161 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.576769    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-v6gdq"]
Nov 03 07:59:27.578161 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.576817    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"]
Nov 03 07:59:27.578897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.578670    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.584367 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.584196    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 07:59:27.584367 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.584332    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2r6wd\""
Nov 03 07:59:27.584746 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.584721    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.591631 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.591606    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"]
Nov 03 07:59:27.614010 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.613977    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f7d0674-9737-4d55-8110-912835303c92-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tgbhc\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.614218 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.614103    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35013bdc-2709-4435-8e0f-cbb535870313-cert\") pod \"kserve-controller-manager-776cc66d47-v6gdq\" (UID: \"35013bdc-2709-4435-8e0f-cbb535870313\") " pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.614218 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.614166    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dxs\" (UniqueName: \"kubernetes.io/projected/9f7d0674-9737-4d55-8110-912835303c92-kube-api-access-k9dxs\") pod \"llmisvc-controller-manager-869d5dcc5d-tgbhc\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.614326 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.614210    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cx9m\" (UniqueName: \"kubernetes.io/projected/35013bdc-2709-4435-8e0f-cbb535870313-kube-api-access-8cx9m\") pod \"kserve-controller-manager-776cc66d47-v6gdq\" (UID: \"35013bdc-2709-4435-8e0f-cbb535870313\") " pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.631343 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.631242    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:27.682862 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.682812    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 07:59:27.682862 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.682835    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-h78zr\""
Nov 03 07:59:27.715505 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.715454    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35013bdc-2709-4435-8e0f-cbb535870313-cert\") pod \"kserve-controller-manager-776cc66d47-v6gdq\" (UID: \"35013bdc-2709-4435-8e0f-cbb535870313\") " pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.715695 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.715527    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dxs\" (UniqueName: \"kubernetes.io/projected/9f7d0674-9737-4d55-8110-912835303c92-kube-api-access-k9dxs\") pod \"llmisvc-controller-manager-869d5dcc5d-tgbhc\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.715695 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.715579    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cx9m\" (UniqueName: \"kubernetes.io/projected/35013bdc-2709-4435-8e0f-cbb535870313-kube-api-access-8cx9m\") pod \"kserve-controller-manager-776cc66d47-v6gdq\" (UID: \"35013bdc-2709-4435-8e0f-cbb535870313\") " pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.715695 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.715645    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f7d0674-9737-4d55-8110-912835303c92-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tgbhc\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.718456 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.718430    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f7d0674-9737-4d55-8110-912835303c92-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tgbhc\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.718456 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.718458    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35013bdc-2709-4435-8e0f-cbb535870313-cert\") pod \"kserve-controller-manager-776cc66d47-v6gdq\" (UID: \"35013bdc-2709-4435-8e0f-cbb535870313\") " pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.725150 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.725112    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cx9m\" (UniqueName: \"kubernetes.io/projected/35013bdc-2709-4435-8e0f-cbb535870313-kube-api-access-8cx9m\") pod \"kserve-controller-manager-776cc66d47-v6gdq\" (UID: \"35013bdc-2709-4435-8e0f-cbb535870313\") " pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.726165 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.726118    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dxs\" (UniqueName: \"kubernetes.io/projected/9f7d0674-9737-4d55-8110-912835303c92-kube-api-access-k9dxs\") pod \"llmisvc-controller-manager-869d5dcc5d-tgbhc\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.760900 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.760814    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:27.768333 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.768296    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"]
Nov 03 07:59:27.778602 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.778576    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.792096 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.788243    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"]
Nov 03 07:59:27.816924 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.816897    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.817102 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.816951    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.817102 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.817046    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.817238 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.817164    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.817238 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.817225    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5943db42-3998-4ec5-87f2-77cbda6b5c07-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.817358 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.817260    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26fd\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-kube-api-access-g26fd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.817358 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.817330    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.903123 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.903099    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:27.912964 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.912938    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:27.918052 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.917950    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5943db42-3998-4ec5-87f2-77cbda6b5c07-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.918052 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.917977    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26fd\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-kube-api-access-g26fd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.918052 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.918012    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.918254 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.918097    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.918254 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.918149    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.918254 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.918174    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.918409 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.918283    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.919315 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.918950    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.920940 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.920911    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5943db42-3998-4ec5-87f2-77cbda6b5c07-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.921897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.921690    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.921897 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.921787    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.922106 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.922084    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.940980 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.940903    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.941751 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.941686    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26fd\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-kube-api-access-g26fd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z6qn\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:27.961291 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:27.961257    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:28.077438 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.077405    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:28.109838 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.109803    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:28.198105 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.198055    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-v6gdq"]
Nov 03 07:59:28.206310 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.206014    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"]
Nov 03 07:59:28.211805 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:59:28.211771    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f7d0674_9737_4d55_8110_912835303c92.slice/crio-3646fb3f7d223f40786b64b7958a72c29aaaf9363ad215bcec97393b024891b3 WatchSource:0}: Error finding container 3646fb3f7d223f40786b64b7958a72c29aaaf9363ad215bcec97393b024891b3: Status 404 returned error can't find the container with id 3646fb3f7d223f40786b64b7958a72c29aaaf9363ad215bcec97393b024891b3
Nov 03 07:59:28.380533 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:59:28.380451    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 07:59:28.403196 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.403161    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"]
Nov 03 07:59:28.407401 ip-10-0-140-232 kubenswrapper[6430]: W1103 07:59:28.407366    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5943db42_3998_4ec5_87f2_77cbda6b5c07.slice/crio-8b8079b4374c0109dc047bc0ac8d3c607cf04df2e9668f54db9440a6ed538392 WatchSource:0}: Error finding container 8b8079b4374c0109dc047bc0ac8d3c607cf04df2e9668f54db9440a6ed538392: Status 404 returned error can't find the container with id 8b8079b4374c0109dc047bc0ac8d3c607cf04df2e9668f54db9440a6ed538392
Nov 03 07:59:28.409525 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.409491    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 07:59:28.409608 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.409564    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 07:59:28.551928 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.551884    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:28.660655 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.660555    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:28.768557 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.768525    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:28.808282 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.808242    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq" event={"ID":"21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe","Type":"ContainerStarted","Data":"396e788668a5c261aa61e7ad9a074b17711a109e703dad081bc179554fc7381a"}
Nov 03 07:59:28.808725 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.808704    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:28.810769 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.810736    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq" event={"ID":"35013bdc-2709-4435-8e0f-cbb535870313","Type":"ContainerStarted","Data":"2acabdeb4c3f522bbe9cf728da2e0c05618fd8f2f0a1d565b6eb8daeaa72ced4"}
Nov 03 07:59:28.813578 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.813480    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc" event={"ID":"9f7d0674-9737-4d55-8110-912835303c92","Type":"ContainerStarted","Data":"3646fb3f7d223f40786b64b7958a72c29aaaf9363ad215bcec97393b024891b3"}
Nov 03 07:59:28.815836 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.815808    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn" event={"ID":"5943db42-3998-4ec5-87f2-77cbda6b5c07","Type":"ContainerStarted","Data":"991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262"}
Nov 03 07:59:28.815972 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.815858    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn" event={"ID":"5943db42-3998-4ec5-87f2-77cbda6b5c07","Type":"ContainerStarted","Data":"8b8079b4374c0109dc047bc0ac8d3c607cf04df2e9668f54db9440a6ed538392"}
Nov 03 07:59:28.815972 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.815962    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:28.845399 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.845330    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq" podStartSLOduration=1.8797811279999999 podStartE2EDuration="8.845309555s" totalImagesPullingTime="6.965528427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:28.826276271 +0000 UTC m=+318.329111793" watchObservedRunningTime="2026-11-03 07:59:28.845309555 +0000 UTC m=+318.348145077"
Nov 03 07:59:28.845583 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:28.845446    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn" podStartSLOduration=1.8454408180000001 podStartE2EDuration="1.845440818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:28.84441382 +0000 UTC m=+318.347249342" watchObservedRunningTime="2026-11-03 07:59:28.845440818 +0000 UTC m=+318.348276337"
Nov 03 07:59:29.822831 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:29.822775    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 07:59:29.865595 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:29.865571    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"]
Nov 03 07:59:29.865761 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:29.865741    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp" podUID="acea5f02-57a7-42cf-8a6f-808d02af7049" containerName="discovery" containerID="cri-o://028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a" gracePeriod=30
Nov 03 07:59:30.114602 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.114579    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:59:30.140663 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.140627    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-cacerts\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-cacerts\") pod \"acea5f02-57a7-42cf-8a6f-808d02af7049\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") "
Nov 03 07:59:30.140813 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.140672    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-kube-api-access-kjmkr\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-kube-api-access-kjmkr\") pod \"acea5f02-57a7-42cf-8a6f-808d02af7049\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") "
Nov 03 07:59:30.140813 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.140698    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-dns-cert\") pod \"acea5f02-57a7-42cf-8a6f-808d02af7049\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") "
Nov 03 07:59:30.140813 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.140741    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-ca-configmap\") pod \"acea5f02-57a7-42cf-8a6f-808d02af7049\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") "
Nov 03 07:59:30.140813 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.140779    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-token\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-token\") pod \"acea5f02-57a7-42cf-8a6f-808d02af7049\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") "
Nov 03 07:59:30.141092 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.140813    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acea5f02-57a7-42cf-8a6f-808d02af7049-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/acea5f02-57a7-42cf-8a6f-808d02af7049-local-certs\") pod \"acea5f02-57a7-42cf-8a6f-808d02af7049\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") "
Nov 03 07:59:30.141092 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.140897    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-kubeconfig\") pod \"acea5f02-57a7-42cf-8a6f-808d02af7049\" (UID: \"acea5f02-57a7-42cf-8a6f-808d02af7049\") "
Nov 03 07:59:30.141269 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.141225    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-ca-configmap" pod "acea5f02-57a7-42cf-8a6f-808d02af7049" (UID: "acea5f02-57a7-42cf-8a6f-808d02af7049"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:59:30.143815 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.143584    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-cacerts" pod "acea5f02-57a7-42cf-8a6f-808d02af7049" (UID: "acea5f02-57a7-42cf-8a6f-808d02af7049"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:59:30.143815 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.143736    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-kube-api-access-kjmkr" pod "acea5f02-57a7-42cf-8a6f-808d02af7049" (UID: "acea5f02-57a7-42cf-8a6f-808d02af7049"). InnerVolumeSpecName "kube-api-access-kjmkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:59:30.144003 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.143909    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-dns-cert" pod "acea5f02-57a7-42cf-8a6f-808d02af7049" (UID: "acea5f02-57a7-42cf-8a6f-808d02af7049"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:59:30.144085 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.144013    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-token" pod "acea5f02-57a7-42cf-8a6f-808d02af7049" (UID: "acea5f02-57a7-42cf-8a6f-808d02af7049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:59:30.144307 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.144264    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-kubeconfig" pod "acea5f02-57a7-42cf-8a6f-808d02af7049" (UID: "acea5f02-57a7-42cf-8a6f-808d02af7049"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:59:30.144487 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.144417    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acea5f02-57a7-42cf-8a6f-808d02af7049-local-certs" pod "acea5f02-57a7-42cf-8a6f-808d02af7049" (UID: "acea5f02-57a7-42cf-8a6f-808d02af7049"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:59:30.241905 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.241873    6430 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-cacerts\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:30.241905 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.241905    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjmkr\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-kube-api-access-kjmkr\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:30.242130 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.241916    6430 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-dns-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:30.242130 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.241927    6430 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-csr-ca-configmap\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:30.242130 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.241936    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:30.242130 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.241945    6430 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/acea5f02-57a7-42cf-8a6f-808d02af7049-local-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:30.242130 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.241953    6430 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acea5f02-57a7-42cf-8a6f-808d02af7049-istio-kubeconfig\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:59:30.826946 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.826903    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx" event={"ID":"3c8d6356-3592-4952-846d-3b7c992b6a3c","Type":"ContainerStarted","Data":"bbdb61e47ee55e45c304dcb535eece97c8a41dcaf8099ef057ea66d74de4cdac"}
Nov 03 07:59:30.828434 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.828404    6430 generic.go:356] "Generic (PLEG): container finished" podID="acea5f02-57a7-42cf-8a6f-808d02af7049" containerID="028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a" exitCode=0
Nov 03 07:59:30.828571 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.828446    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"
Nov 03 07:59:30.828571 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.828486    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp" event={"ID":"acea5f02-57a7-42cf-8a6f-808d02af7049","Type":"ContainerDied","Data":"028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a"}
Nov 03 07:59:30.828571 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.828529    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp" event={"ID":"acea5f02-57a7-42cf-8a6f-808d02af7049","Type":"ContainerDied","Data":"d81f02562530f9ee7e5e6315ef92c2006f0744deef7f8744898d15804c2cf3d8"}
Nov 03 07:59:30.828571 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.828551    6430 scope.go:122] "RemoveContainer" containerID="028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a"
Nov 03 07:59:30.838684 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.838656    6430 scope.go:122] "RemoveContainer" containerID="028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a"
Nov 03 07:59:30.839107 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:59:30.839056    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a\": container with ID starting with 028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a not found: ID does not exist" containerID="028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a"
Nov 03 07:59:30.839203 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.839117    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a"} err="failed to get container status \"028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a\": rpc error: code = NotFound desc = could not find container \"028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a\": container with ID starting with 028c8b44e43ad425b3a7734037d1b7756c7103269f2d81cf0962e09d3ebd009a not found: ID does not exist"
Nov 03 07:59:30.846597 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.846552    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qp7tx" podStartSLOduration=1.720944071 podStartE2EDuration="12.846538421s" totalImagesPullingTime="11.12559435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:30.845202561 +0000 UTC m=+320.348038079" watchObservedRunningTime="2026-11-03 07:59:30.846538421 +0000 UTC m=+320.349373944"
Nov 03 07:59:30.870685 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.868382    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"]
Nov 03 07:59:30.870685 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:30.870512    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4jgnp"]
Nov 03 07:59:31.835825 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:31.835789    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq" event={"ID":"35013bdc-2709-4435-8e0f-cbb535870313","Type":"ContainerStarted","Data":"6e40c0f61a6a3b0866ee2a4b4e8334cfcd5e3ebc60909dbe769d315a9c30ae08"}
Nov 03 07:59:31.836312 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:31.835921    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 07:59:31.837331 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:31.837306    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc" event={"ID":"9f7d0674-9737-4d55-8110-912835303c92","Type":"ContainerStarted","Data":"661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176"}
Nov 03 07:59:31.837501 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:31.837486    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 07:59:31.850761 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:31.850686    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq" podStartSLOduration=1.9450421119999999 podStartE2EDuration="4.85067s" totalImagesPullingTime="2.905627888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:31.850295253 +0000 UTC m=+321.353130769" watchObservedRunningTime="2026-11-03 07:59:31.85067 +0000 UTC m=+321.353505513"
Nov 03 07:59:31.864803 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:31.864635    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc" podStartSLOduration=1.479437503 podStartE2EDuration="4.864622624s" totalImagesPullingTime="3.385185121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:31.863613353 +0000 UTC m=+321.366448868" watchObservedRunningTime="2026-11-03 07:59:31.864622624 +0000 UTC m=+321.367458138"
Nov 03 07:59:32.616931 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:32.616901    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acea5f02-57a7-42cf-8a6f-808d02af7049" path="/var/lib/kubelet/pods/acea5f02-57a7-42cf-8a6f-808d02af7049/volumes"
Nov 03 07:59:34.850084 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:34.850040    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t88sq"
Nov 03 07:59:35.390813 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:35.390777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:35.500287 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:35.500257    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:35.606571 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:35.606539    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:41.551229 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:41.551201    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:41.660468 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:41.660432    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:41.762561 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:41.762523    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:41.872113 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:41.871995    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:46.016639 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:46.016596    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:46.194533 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:46.194497    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:47.608077 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:47.608035    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:47.719868 ip-10-0-140-232 kubenswrapper[6430]: I1103 07:59:47.719821    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 07:59:58.373845 ip-10-0-140-232 kubenswrapper[6430]: E1103 07:59:58.373806    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:00:02.843814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:02.843783    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 08:00:02.846714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:02.846696    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-v6gdq"
Nov 03 08:00:11.227498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:11.227452    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:16.143222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:16.143183    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:16.262842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:16.262808    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:17.509419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.509382    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:17.656698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.656662    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:17.719508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.719470    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"]
Nov 03 08:00:17.719860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.719845    6430 cpu_manager.go:414] "containerMap: removing container" podUID="acea5f02-57a7-42cf-8a6f-808d02af7049" containerName="discovery"
Nov 03 08:00:17.719908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.719862    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acea5f02-57a7-42cf-8a6f-808d02af7049" containerName="discovery"
Nov 03 08:00:17.719945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.719930    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="acea5f02-57a7-42cf-8a6f-808d02af7049" containerName="discovery"
Nov 03 08:00:17.759261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.759230    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"]
Nov 03 08:00:17.759403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.759337    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.762587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.762525    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 08:00:17.762714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.762607    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7nz66\""
Nov 03 08:00:17.762714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.762612    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 08:00:17.778201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.778176    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:17.835316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.835293    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4ts\" (UniqueName: \"kubernetes.io/projected/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-kube-api-access-6p4ts\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.835416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.835322    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-metrics-certs\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.835416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.835342    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-maas-api-tls\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.835506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.835457    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-openshift-service-ca\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.936124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.936105    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4ts\" (UniqueName: \"kubernetes.io/projected/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-kube-api-access-6p4ts\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.936220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.936134    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-metrics-certs\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.936220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.936163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-maas-api-tls\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.936220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.936213    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-openshift-service-ca\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.936779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.936762    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-openshift-service-ca\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.938592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.938572    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-metrics-certs\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.938680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.938649    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-maas-api-tls\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:17.943304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:17.943281    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4ts\" (UniqueName: \"kubernetes.io/projected/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-kube-api-access-6p4ts\") pod \"maas-api-6d669c7fcd-44pfm\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:18.072103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:18.072018    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:18.344549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:18.344472    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"]
Nov 03 08:00:18.348172 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:00:18.348143    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4376152_023c_4aac_aa34_4a1c9c9fd8d0.slice/crio-5c45b2918587c3298ca883cfb7f529c4b0a7deb6da4c6ce6e004a10efe7811b8 WatchSource:0}: Error finding container 5c45b2918587c3298ca883cfb7f529c4b0a7deb6da4c6ce6e004a10efe7811b8: Status 404 returned error can't find the container with id 5c45b2918587c3298ca883cfb7f529c4b0a7deb6da4c6ce6e004a10efe7811b8
Nov 03 08:00:19.031174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:19.031131    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm" event={"ID":"a4376152-023c-4aac-aa34-4a1c9c9fd8d0","Type":"ContainerStarted","Data":"5c45b2918587c3298ca883cfb7f529c4b0a7deb6da4c6ce6e004a10efe7811b8"}
Nov 03 08:00:22.046394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:22.046355    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm" event={"ID":"a4376152-023c-4aac-aa34-4a1c9c9fd8d0","Type":"ContainerStarted","Data":"4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67"}
Nov 03 08:00:22.046786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:22.046508    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:22.061487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:22.061434    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm" podStartSLOduration=1.512422836 podStartE2EDuration="5.0614179s" totalImagesPullingTime="3.548995064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:22.059942323 +0000 UTC m=+371.562777838" watchObservedRunningTime="2026-11-03 08:00:22.0614179 +0000 UTC m=+371.564253416"
Nov 03 08:00:28.055723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:28.055695    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:28.373613 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:00:28.373528    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:00:28.494350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:28.494316    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:28.673772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:28.673688    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:00:32.118301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.118267    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"]
Nov 03 08:00:32.121781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.121761    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.131683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.131658    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"]
Nov 03 08:00:32.159296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.159262    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/862e4682-de0b-4e34-9bb8-626f0ab19a83-cert\") pod \"llmisvc-controller-manager-6466c554d8-r57t2\" (UID: \"862e4682-de0b-4e34-9bb8-626f0ab19a83\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.159543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.159309    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrps8\" (UniqueName: \"kubernetes.io/projected/862e4682-de0b-4e34-9bb8-626f0ab19a83-kube-api-access-mrps8\") pod \"llmisvc-controller-manager-6466c554d8-r57t2\" (UID: \"862e4682-de0b-4e34-9bb8-626f0ab19a83\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.260507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.260473    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/862e4682-de0b-4e34-9bb8-626f0ab19a83-cert\") pod \"llmisvc-controller-manager-6466c554d8-r57t2\" (UID: \"862e4682-de0b-4e34-9bb8-626f0ab19a83\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.260663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.260514    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrps8\" (UniqueName: \"kubernetes.io/projected/862e4682-de0b-4e34-9bb8-626f0ab19a83-kube-api-access-mrps8\") pod \"llmisvc-controller-manager-6466c554d8-r57t2\" (UID: \"862e4682-de0b-4e34-9bb8-626f0ab19a83\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.263251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.263224    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/862e4682-de0b-4e34-9bb8-626f0ab19a83-cert\") pod \"llmisvc-controller-manager-6466c554d8-r57t2\" (UID: \"862e4682-de0b-4e34-9bb8-626f0ab19a83\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.267563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.267540    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrps8\" (UniqueName: \"kubernetes.io/projected/862e4682-de0b-4e34-9bb8-626f0ab19a83-kube-api-access-mrps8\") pod \"llmisvc-controller-manager-6466c554d8-r57t2\" (UID: \"862e4682-de0b-4e34-9bb8-626f0ab19a83\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.434140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.434107    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:32.705471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:32.705346    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"]
Nov 03 08:00:32.709201 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:00:32.709169    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862e4682_de0b_4e34_9bb8_626f0ab19a83.slice/crio-a3de3292fa0235e1414055b53072d12852e24777ac71321efb5af16aed2273d2 WatchSource:0}: Error finding container a3de3292fa0235e1414055b53072d12852e24777ac71321efb5af16aed2273d2: Status 404 returned error can't find the container with id a3de3292fa0235e1414055b53072d12852e24777ac71321efb5af16aed2273d2
Nov 03 08:00:33.097372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:33.097285    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2" event={"ID":"862e4682-de0b-4e34-9bb8-626f0ab19a83","Type":"ContainerStarted","Data":"a3de3292fa0235e1414055b53072d12852e24777ac71321efb5af16aed2273d2"}
Nov 03 08:00:34.102756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:34.102720    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2" event={"ID":"862e4682-de0b-4e34-9bb8-626f0ab19a83","Type":"ContainerStarted","Data":"2578e98012e07005c4969e0ac6a2d2a7265dc5e7edd2f42255ba18f0d0e87335"}
Nov 03 08:00:34.103229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:34.102852    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:00:34.116939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:34.116896    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2" podStartSLOduration=1.547387055 podStartE2EDuration="2.116883621s" totalImagesPullingTime="569.496566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:34.116232161 +0000 UTC m=+383.619067678" watchObservedRunningTime="2026-11-03 08:00:34.116883621 +0000 UTC m=+383.619719176"
Nov 03 08:00:39.831369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:39.831343    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"]
Nov 03 08:00:39.831771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:39.831541    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm" podUID="a4376152-023c-4aac-aa34-4a1c9c9fd8d0" containerName="maas-api" containerID="cri-o://4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67" gracePeriod=30
Nov 03 08:00:40.070079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.070034    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:40.122573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.122489    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-metrics-certs\") pod \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") "
Nov 03 08:00:40.122573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.122523    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-maas-api-tls\") pod \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") "
Nov 03 08:00:40.122573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.122549    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-openshift-service-ca\") pod \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") "
Nov 03 08:00:40.122849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.122640    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-kube-api-access-6p4ts\" (UniqueName: \"kubernetes.io/projected/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-kube-api-access-6p4ts\") pod \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\" (UID: \"a4376152-023c-4aac-aa34-4a1c9c9fd8d0\") "
Nov 03 08:00:40.122945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.122916    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-openshift-service-ca" pod "a4376152-023c-4aac-aa34-4a1c9c9fd8d0" (UID: "a4376152-023c-4aac-aa34-4a1c9c9fd8d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:00:40.125141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.125118    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-metrics-certs" pod "a4376152-023c-4aac-aa34-4a1c9c9fd8d0" (UID: "a4376152-023c-4aac-aa34-4a1c9c9fd8d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:00:40.125351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.125331    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-maas-api-tls" pod "a4376152-023c-4aac-aa34-4a1c9c9fd8d0" (UID: "a4376152-023c-4aac-aa34-4a1c9c9fd8d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:00:40.125351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.125333    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-kube-api-access-6p4ts" pod "a4376152-023c-4aac-aa34-4a1c9c9fd8d0" (UID: "a4376152-023c-4aac-aa34-4a1c9c9fd8d0"). InnerVolumeSpecName "kube-api-access-6p4ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:00:40.127853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.127828    6430 generic.go:356] "Generic (PLEG): container finished" podID="a4376152-023c-4aac-aa34-4a1c9c9fd8d0" containerID="4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67" exitCode=0
Nov 03 08:00:40.127949 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.127866    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm" event={"ID":"a4376152-023c-4aac-aa34-4a1c9c9fd8d0","Type":"ContainerDied","Data":"4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67"}
Nov 03 08:00:40.127949 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.127878    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"
Nov 03 08:00:40.127949 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.127909    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm" event={"ID":"a4376152-023c-4aac-aa34-4a1c9c9fd8d0","Type":"ContainerDied","Data":"5c45b2918587c3298ca883cfb7f529c4b0a7deb6da4c6ce6e004a10efe7811b8"}
Nov 03 08:00:40.127949 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.127931    6430 scope.go:122] "RemoveContainer" containerID="4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67"
Nov 03 08:00:40.141696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.141659    6430 scope.go:122] "RemoveContainer" containerID="4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67"
Nov 03 08:00:40.141982 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:00:40.141964    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67\": container with ID starting with 4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67 not found: ID does not exist" containerID="4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67"
Nov 03 08:00:40.142037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.141990    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67"} err="failed to get container status \"4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67\": rpc error: code = NotFound desc = could not find container \"4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67\": container with ID starting with 4338e867a82243f257fff26ae4dca4de7d657370e2f072766397bd0f29b51a67 not found: ID does not exist"
Nov 03 08:00:40.149476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.149456    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"]
Nov 03 08:00:40.154523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.154499    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-44pfm"]
Nov 03 08:00:40.223383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.223350    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4ts\" (UniqueName: \"kubernetes.io/projected/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-kube-api-access-6p4ts\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:40.223383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.223379    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:40.223572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.223393    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:40.223572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.223406    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4376152-023c-4aac-aa34-4a1c9c9fd8d0-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:40.615126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:00:40.615097    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4376152-023c-4aac-aa34-4a1c9c9fd8d0" path="/var/lib/kubelet/pods/a4376152-023c-4aac-aa34-4a1c9c9fd8d0/volumes"
Nov 03 08:00:58.373887 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:00:58.373849    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:01:05.109187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.109159    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-r57t2"
Nov 03 08:01:05.149690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.149659    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"]
Nov 03 08:01:05.150018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.149959    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc" podUID="9f7d0674-9737-4d55-8110-912835303c92" containerName="manager" containerID="cri-o://661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176" gracePeriod=30
Nov 03 08:01:05.397023 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.396994    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 08:01:05.533555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.533522    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7d0674-9737-4d55-8110-912835303c92-kube-api-access-k9dxs\" (UniqueName: \"kubernetes.io/projected/9f7d0674-9737-4d55-8110-912835303c92-kube-api-access-k9dxs\") pod \"9f7d0674-9737-4d55-8110-912835303c92\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") "
Nov 03 08:01:05.533736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.533591    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f7d0674-9737-4d55-8110-912835303c92-cert\" (UniqueName: \"kubernetes.io/secret/9f7d0674-9737-4d55-8110-912835303c92-cert\") pod \"9f7d0674-9737-4d55-8110-912835303c92\" (UID: \"9f7d0674-9737-4d55-8110-912835303c92\") "
Nov 03 08:01:05.535818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.535789    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7d0674-9737-4d55-8110-912835303c92-cert" pod "9f7d0674-9737-4d55-8110-912835303c92" (UID: "9f7d0674-9737-4d55-8110-912835303c92"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:01:05.535953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.535839    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7d0674-9737-4d55-8110-912835303c92-kube-api-access-k9dxs" pod "9f7d0674-9737-4d55-8110-912835303c92" (UID: "9f7d0674-9737-4d55-8110-912835303c92"). InnerVolumeSpecName "kube-api-access-k9dxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:01:05.635051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.634971    6430 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f7d0674-9737-4d55-8110-912835303c92-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:01:05.635051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.635003    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dxs\" (UniqueName: \"kubernetes.io/projected/9f7d0674-9737-4d55-8110-912835303c92-kube-api-access-k9dxs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:01:05.981036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:05.980999    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:06.095652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.095619    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:06.247561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.247471    6430 generic.go:356] "Generic (PLEG): container finished" podID="9f7d0674-9737-4d55-8110-912835303c92" containerID="661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176" exitCode=0
Nov 03 08:01:06.247561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.247531    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"
Nov 03 08:01:06.248039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.247552    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc" event={"ID":"9f7d0674-9737-4d55-8110-912835303c92","Type":"ContainerDied","Data":"661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176"}
Nov 03 08:01:06.248039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.247608    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc" event={"ID":"9f7d0674-9737-4d55-8110-912835303c92","Type":"ContainerDied","Data":"3646fb3f7d223f40786b64b7958a72c29aaaf9363ad215bcec97393b024891b3"}
Nov 03 08:01:06.248039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.247630    6430 scope.go:122] "RemoveContainer" containerID="661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176"
Nov 03 08:01:06.256276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.256262    6430 scope.go:122] "RemoveContainer" containerID="661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176"
Nov 03 08:01:06.256529 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:01:06.256510    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176\": container with ID starting with 661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176 not found: ID does not exist" containerID="661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176"
Nov 03 08:01:06.256565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.256539    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176"} err="failed to get container status \"661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176\": rpc error: code = NotFound desc = could not find container \"661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176\": container with ID starting with 661481fd1b2dbbb10bd2fb758b6949ee173ea773469fd48c7db5e28567d1f176 not found: ID does not exist"
Nov 03 08:01:06.266156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.266129    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"]
Nov 03 08:01:06.267864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.267844    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tgbhc"]
Nov 03 08:01:06.615764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:06.615685    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7d0674-9737-4d55-8110-912835303c92" path="/var/lib/kubelet/pods/9f7d0674-9737-4d55-8110-912835303c92/volumes"
Nov 03 08:01:11.269261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:11.269227    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:22.316290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:22.316217    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:25.511934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:25.511877    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:26.359023 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:26.358993    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:26.878279 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:26.878233    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:28.374149 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:01:28.374114    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:01:29.584749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:29.584717    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:30.429433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:30.429394    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:31.670758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:31.670725    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:33.294759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:33.294728    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:50.352123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:50.352025    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:50.457851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:50.457818    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:51.181081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.181030    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"]
Nov 03 08:01:51.181532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.181510    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9f7d0674-9737-4d55-8110-912835303c92" containerName="manager"
Nov 03 08:01:51.181665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.181534    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7d0674-9737-4d55-8110-912835303c92" containerName="manager"
Nov 03 08:01:51.181665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.181550    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a4376152-023c-4aac-aa34-4a1c9c9fd8d0" containerName="maas-api"
Nov 03 08:01:51.181665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.181557    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4376152-023c-4aac-aa34-4a1c9c9fd8d0" containerName="maas-api"
Nov 03 08:01:51.181665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.181646    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7d0674-9737-4d55-8110-912835303c92" containerName="manager"
Nov 03 08:01:51.181665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.181659    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4376152-023c-4aac-aa34-4a1c9c9fd8d0" containerName="maas-api"
Nov 03 08:01:51.184596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.184574    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.200759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.200729    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"]
Nov 03 08:01:51.275772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.275732    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zg2vp\""
Nov 03 08:01:51.310137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310104    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310176    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310268    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310290    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b27641-47ac-4a12-9a06-96b6b02240b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxgts\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-kube-api-access-lxgts\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310331    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310391    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.310502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.310467    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411385    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411427    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411443    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411491    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411525    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b27641-47ac-4a12-9a06-96b6b02240b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411550    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxgts\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-kube-api-access-lxgts\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411581    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411606    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.411840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411667    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.412244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411859    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.412244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411909    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.412244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411993    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.412244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.411995    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.412542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.412521    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b27641-47ac-4a12-9a06-96b6b02240b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.414026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.414004    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.414126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.414037    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.418934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.418910    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxgts\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-kube-api-access-lxgts\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.418934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.418926    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bm2g7\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.498408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.498360    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:01:51.788640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:51.788559    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"]
Nov 03 08:01:51.792439 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:01:51.792411    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b27641_47ac_4a12_9a06_96b6b02240b8.slice/crio-76e1da6336a77bac3685e53c37b3548858a52848433691a44c04fe056ecc3646 WatchSource:0}: Error finding container 76e1da6336a77bac3685e53c37b3548858a52848433691a44c04fe056ecc3646: Status 404 returned error can't find the container with id 76e1da6336a77bac3685e53c37b3548858a52848433691a44c04fe056ecc3646
Nov 03 08:01:52.444726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:52.444691    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" event={"ID":"08b27641-47ac-4a12-9a06-96b6b02240b8","Type":"ContainerStarted","Data":"76e1da6336a77bac3685e53c37b3548858a52848433691a44c04fe056ecc3646"}
Nov 03 08:01:55.536749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:55.536713    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:55.652782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:55.652745    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:57.965145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:57.965105    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:01:58.376112 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:01:58.376018    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:01:59.563913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:59.563875    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:01:59.564343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:59.563959    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:01:59.564343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:01:59.564007    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:02:00.479247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:00.479207    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" event={"ID":"08b27641-47ac-4a12-9a06-96b6b02240b8","Type":"ContainerStarted","Data":"c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b"}
Nov 03 08:02:00.500791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:00.500762    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:02:00.507185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:00.507138    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 08:02:00.507457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:00.507418    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 08:02:01.509087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:01.505401    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 08:02:01.509087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:01.505455    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 08:02:02.511437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:02.511401    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 08:02:02.512039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:02.512007    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 08:02:03.519088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:03.511170    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 08:02:03.519088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:03.511272    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 08:02:04.504480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:04.504439    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 08:02:04.504762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:04.504732    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 08:02:05.499494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:05.499454    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 08:02:05.499939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:05.499545    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 08:02:06.202965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.202906    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podStartSLOduration=7.434208877 podStartE2EDuration="15.202890877s" totalImagesPullingTime="7.768682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:00.500467968 +0000 UTC m=+470.003303483" watchObservedRunningTime="2026-11-03 08:02:06.202890877 +0000 UTC m=+475.705726390"
Nov 03 08:02:06.204134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.204109    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"]
Nov 03 08:02:06.208543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.208523    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.220645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.220613    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"]
Nov 03 08:02:06.261147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261102    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261145    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261228    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5a8bac-1e6a-4fb8-9852-abc001854117-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261291    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261331    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f7d\" (UniqueName: \"kubernetes.io/projected/7a5a8bac-1e6a-4fb8-9852-abc001854117-kube-api-access-g6f7d\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261369    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261400    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261466    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.261619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.261560    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.362696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.362651    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.362696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.362702    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.362954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.362761    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.362954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.362797    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.362954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.362823    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.362954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.362851    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5a8bac-1e6a-4fb8-9852-abc001854117-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363024    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363080    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f7d\" (UniqueName: \"kubernetes.io/projected/7a5a8bac-1e6a-4fb8-9852-abc001854117-kube-api-access-g6f7d\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363137    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363516    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363676    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363710    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5a8bac-1e6a-4fb8-9852-abc001854117-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363568    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.363849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.363740    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.365583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.365549    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.365666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.365652    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.371947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.371921    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5a8bac-1e6a-4fb8-9852-abc001854117-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.372148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.372129    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f7d\" (UniqueName: \"kubernetes.io/projected/7a5a8bac-1e6a-4fb8-9852-abc001854117-kube-api-access-g6f7d\") pod \"maas-default-gateway-openshift-default-76f4876667-66kgk\" (UID: \"7a5a8bac-1e6a-4fb8-9852-abc001854117\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.498957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.498867    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:02:06.499191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.499168    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Nov 03 08:02:06.499277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.499241    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Nov 03 08:02:06.521827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.521803    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:06.801933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.801853    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"]
Nov 03 08:02:06.805040 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:02:06.805004    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5a8bac_1e6a_4fb8_9852_abc001854117.slice/crio-7d8a7607f308db8e87d019865f0cf99a1371f61311068b92cee3e5cada9c8022 WatchSource:0}: Error finding container 7d8a7607f308db8e87d019865f0cf99a1371f61311068b92cee3e5cada9c8022: Status 404 returned error can't find the container with id 7d8a7607f308db8e87d019865f0cf99a1371f61311068b92cee3e5cada9c8022
Nov 03 08:02:06.808395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.808352    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:02:06.808484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.808453    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:02:06.808525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:06.808502    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:02:07.540740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:07.540687    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" event={"ID":"7a5a8bac-1e6a-4fb8-9852-abc001854117","Type":"ContainerStarted","Data":"1aed3ac0db92541b244cc3dd8a61547a7e639eb1f2cf8c19141c886aa5a16a80"}
Nov 03 08:02:07.541298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:07.540797    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" event={"ID":"7a5a8bac-1e6a-4fb8-9852-abc001854117","Type":"ContainerStarted","Data":"7d8a7607f308db8e87d019865f0cf99a1371f61311068b92cee3e5cada9c8022"}
Nov 03 08:02:07.557922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:07.557871    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podStartSLOduration=1.557858234 podStartE2EDuration="1.557858234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:07.557729455 +0000 UTC m=+477.060564969" watchObservedRunningTime="2026-11-03 08:02:07.557858234 +0000 UTC m=+477.060693747"
Nov 03 08:02:07.910138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:07.908783    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:02:07.915428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:07.915317    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:02:08.522251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:08.522205    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:08.523969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:08.523944    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-66kgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 08:02:08.524103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:08.524028    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podUID="7a5a8bac-1e6a-4fb8-9852-abc001854117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 08:02:09.522499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:09.522461    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-66kgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 08:02:09.522886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:09.522541    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podUID="7a5a8bac-1e6a-4fb8-9852-abc001854117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 08:02:10.522505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:10.522468    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-66kgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 08:02:10.522903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:10.522538    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podUID="7a5a8bac-1e6a-4fb8-9852-abc001854117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 08:02:11.335791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:11.335742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:11.522472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:11.522433    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-66kgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 08:02:11.522621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:11.522491    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podUID="7a5a8bac-1e6a-4fb8-9852-abc001854117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 08:02:12.522634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:12.522592    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-66kgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 08:02:12.523019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:12.522668    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podUID="7a5a8bac-1e6a-4fb8-9852-abc001854117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 08:02:13.522328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:13.522291    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-66kgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 08:02:13.522500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:13.522347    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podUID="7a5a8bac-1e6a-4fb8-9852-abc001854117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 08:02:15.073748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:15.073712    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:15.074290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:15.074265    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:15.075157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:15.075134    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk"
Nov 03 08:02:19.978921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:19.978865    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:20.454909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:20.454867    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:21.942864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:21.942821    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:22.056908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:22.056851    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:23.369692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:23.369652    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:23.486476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:23.486444    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:28.374328 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:02:28.374295    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:02:44.744736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:44.744679    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:44.850088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:44.850026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:45.200511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:45.200481    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:45.327369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:45.327340    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:45.462766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:45.462673    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:45.624160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:45.624133    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:45.811148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:45.810965    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:45.942103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:45.942043    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:46.164547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:46.164458    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:48.412813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:48.412775    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:48.520655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:48.520605    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:48.925600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:48.925554    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:49.072322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:49.072279    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:49.195945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:49.195920    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:49.355107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:49.355076    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:49.493088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:49.492965    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:49.616403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:49.616372    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:49.931373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:49.931348    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:50.058335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.058306    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:50.763103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.763072    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"]
Nov 03 08:02:50.766322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.766291    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:50.770246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.770224    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.777460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.777433    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"]
Nov 03 08:02:50.783186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.783162    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-6bc553-serving-cert\""
Nov 03 08:02:50.783483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.783461    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 08:02:50.783588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.783519    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7nz66\""
Nov 03 08:02:50.874562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.874523    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-metrics-certs\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.874715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.874662    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmxx\" (UniqueName: \"kubernetes.io/projected/fc5c9dff-5472-4849-add5-af54751535e9-kube-api-access-bjmxx\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.874758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.874717    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c9dff-5472-4849-add5-af54751535e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.874793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.874755    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.881336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.881307    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:50.975481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.975444    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-metrics-certs\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.975677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.975522    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmxx\" (UniqueName: \"kubernetes.io/projected/fc5c9dff-5472-4849-add5-af54751535e9-kube-api-access-bjmxx\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.975677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.975543    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c9dff-5472-4849-add5-af54751535e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.975677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.975569    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.976281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.976224    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c9dff-5472-4849-add5-af54751535e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.978760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.978734    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-metrics-certs\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.978760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.978734    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.983134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.983084    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmxx\" (UniqueName: \"kubernetes.io/projected/fc5c9dff-5472-4849-add5-af54751535e9-kube-api-access-bjmxx\") pod \"maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:50.990602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:50.990538    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:51.084833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.084745    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:51.163829 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.163799    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:51.336156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.326943    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:51.367314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.367009    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"]
Nov 03 08:02:51.371465 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:02:51.371426    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5c9dff_5472_4849_add5_af54751535e9.slice/crio-1bd8c4d47166b85af7ee96bf4be02f0d91b8537d9d3e8158c03b2dc85d7fb0cf WatchSource:0}: Error finding container 1bd8c4d47166b85af7ee96bf4be02f0d91b8537d9d3e8158c03b2dc85d7fb0cf: Status 404 returned error can't find the container with id 1bd8c4d47166b85af7ee96bf4be02f0d91b8537d9d3e8158c03b2dc85d7fb0cf
Nov 03 08:02:51.372980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.372955    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:02:51.726415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.726382    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx" event={"ID":"fc5c9dff-5472-4849-add5-af54751535e9","Type":"ContainerStarted","Data":"1bd8c4d47166b85af7ee96bf4be02f0d91b8537d9d3e8158c03b2dc85d7fb0cf"}
Nov 03 08:02:51.852587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.852552    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"]
Nov 03 08:02:51.857436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.857413    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:51.865865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.865842    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-3e88b7-serving-cert\""
Nov 03 08:02:51.871308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.867468    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"]
Nov 03 08:02:51.952983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.952947    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:51.984962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.984884    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:51.984962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.984940    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79c787b-4338-4a84-9433-ead2be648e61-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:51.985183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.985095    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-metrics-certs\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:51.985183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:51.985132    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwj4h\" (UniqueName: \"kubernetes.io/projected/f79c787b-4338-4a84-9433-ead2be648e61-kube-api-access-kwj4h\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.076976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.076944    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:52.085609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.085577    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79c787b-4338-4a84-9433-ead2be648e61-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.085790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.085631    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-metrics-certs\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.085790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.085653    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwj4h\" (UniqueName: \"kubernetes.io/projected/f79c787b-4338-4a84-9433-ead2be648e61-kube-api-access-kwj4h\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.085790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.085764    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.086519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.086489    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79c787b-4338-4a84-9433-ead2be648e61-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.089530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.089482    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-metrics-certs\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.089743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.089725    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.092973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.092933    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwj4h\" (UniqueName: \"kubernetes.io/projected/f79c787b-4338-4a84-9433-ead2be648e61-kube-api-access-kwj4h\") pod \"maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.182201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.182169    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:52.201662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.201635    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:52.483678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.483648    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"]
Nov 03 08:02:52.487472 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:02:52.487434    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79c787b_4338_4a84_9433_ead2be648e61.slice/crio-6b870685802345da480c14c4056a038892f3b85677a5f03594f928d7a8cba742 WatchSource:0}: Error finding container 6b870685802345da480c14c4056a038892f3b85677a5f03594f928d7a8cba742: Status 404 returned error can't find the container with id 6b870685802345da480c14c4056a038892f3b85677a5f03594f928d7a8cba742
Nov 03 08:02:52.731753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.731705    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx" event={"ID":"fc5c9dff-5472-4849-add5-af54751535e9","Type":"ContainerStarted","Data":"a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7"}
Nov 03 08:02:52.731929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.731890    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:52.732991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.732964    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb" event={"ID":"f79c787b-4338-4a84-9433-ead2be648e61","Type":"ContainerStarted","Data":"6b870685802345da480c14c4056a038892f3b85677a5f03594f928d7a8cba742"}
Nov 03 08:02:52.736187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.736163    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:52.749285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.749245    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx" podStartSLOduration=2.281244855 podStartE2EDuration="2.749232027s" totalImagesPullingTime="467.987172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:52.747760526 +0000 UTC m=+522.250596041" watchObservedRunningTime="2026-11-03 08:02:52.749232027 +0000 UTC m=+522.252067546"
Nov 03 08:02:52.850878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:52.850848    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:53.037508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.037466    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:53.160121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.160050    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:53.280483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.280402    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:53.738474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.738438    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb" event={"ID":"f79c787b-4338-4a84-9433-ead2be648e61","Type":"ContainerStarted","Data":"d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663"}
Nov 03 08:02:53.753712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.753661    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb" podStartSLOduration=2.305379849 podStartE2EDuration="2.753649764s" totalImagesPullingTime="448.269915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:53.753029011 +0000 UTC m=+523.255864525" watchObservedRunningTime="2026-11-03 08:02:53.753649764 +0000 UTC m=+523.256485277"
Nov 03 08:02:53.883440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.883408    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"]
Nov 03 08:02:53.886914 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.886892    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:53.890516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.890445    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-76ef13-serving-cert\""
Nov 03 08:02:53.898040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.898018    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"]
Nov 03 08:02:53.906854 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.906833    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:53.911268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.911245    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmm4\" (UniqueName: \"kubernetes.io/projected/8857534d-237b-4ff9-bf5b-b18cf35727f8-kube-api-access-4zmm4\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:53.911350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.911316    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:53.911350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.911341    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8857534d-237b-4ff9-bf5b-b18cf35727f8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:53.911611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:53.911581    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-metrics-certs\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.012614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.012549    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmm4\" (UniqueName: \"kubernetes.io/projected/8857534d-237b-4ff9-bf5b-b18cf35727f8-kube-api-access-4zmm4\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.012723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.012610    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.012723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.012644    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8857534d-237b-4ff9-bf5b-b18cf35727f8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.012723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.012718    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-metrics-certs\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.012825 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:02:54.012788    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-serving-cert: secret "maas-api-e2e-worker-w0-76ef13-serving-cert" not found
Nov 03 08:02:54.012901 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:02:54.012889    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls podName:8857534d-237b-4ff9-bf5b-b18cf35727f8 nodeName:}" failed. No retries permitted until 2026-11-03 08:02:54.512868504 +0000 UTC m=+524.015704017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls") pod "maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h" (UID: "8857534d-237b-4ff9-bf5b-b18cf35727f8") : secret "maas-api-e2e-worker-w0-76ef13-serving-cert" not found
Nov 03 08:02:54.013458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.013442    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8857534d-237b-4ff9-bf5b-b18cf35727f8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.015235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.015220    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-metrics-certs\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.024136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.024115    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmm4\" (UniqueName: \"kubernetes.io/projected/8857534d-237b-4ff9-bf5b-b18cf35727f8-kube-api-access-4zmm4\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.030205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.030174    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:54.178535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.178500    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:54.516929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.516896    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.519647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.519618    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:54.743081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.743035    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:02:54.805678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:54.802244    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:55.100383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.100302    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"]
Nov 03 08:02:55.105075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.105033    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.114019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.113996    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"]
Nov 03 08:02:55.117952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.117918    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"]
Nov 03 08:02:55.122839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.122812    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-maas-api-tls\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.123013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.122851    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvsk\" (UniqueName: \"kubernetes.io/projected/f0c7df50-1f48-42f9-b524-52ad120683ab-kube-api-access-dgvsk\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.123013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.122901    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0c7df50-1f48-42f9-b524-52ad120683ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.123013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.122945    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-metrics-certs\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.160164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.160136    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-19bb0a-serving-cert\""
Nov 03 08:02:55.195379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.195350    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:55.223493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.223464    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-metrics-certs\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.223643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.223542    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-maas-api-tls\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.223643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.223568    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvsk\" (UniqueName: \"kubernetes.io/projected/f0c7df50-1f48-42f9-b524-52ad120683ab-kube-api-access-dgvsk\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.223643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.223605    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0c7df50-1f48-42f9-b524-52ad120683ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.224343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.224324    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0c7df50-1f48-42f9-b524-52ad120683ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.226049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.226029    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-metrics-certs\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.226170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.226134    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-maas-api-tls\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.230304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.230282    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvsk\" (UniqueName: \"kubernetes.io/projected/f0c7df50-1f48-42f9-b524-52ad120683ab-kube-api-access-dgvsk\") pod \"maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.315147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.315113    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:55.421396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.421318    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:55.696637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.696609    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"]
Nov 03 08:02:55.701099 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:02:55.700376    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c7df50_1f48_42f9_b524_52ad120683ab.slice/crio-c9ab0ffe801b1d590f563dbd5336fc3ebae513c2eeceeda39f7a99a5887eb50e WatchSource:0}: Error finding container c9ab0ffe801b1d590f563dbd5336fc3ebae513c2eeceeda39f7a99a5887eb50e: Status 404 returned error can't find the container with id c9ab0ffe801b1d590f563dbd5336fc3ebae513c2eeceeda39f7a99a5887eb50e
Nov 03 08:02:55.748459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.748429    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv" event={"ID":"f0c7df50-1f48-42f9-b524-52ad120683ab","Type":"ContainerStarted","Data":"c9ab0ffe801b1d590f563dbd5336fc3ebae513c2eeceeda39f7a99a5887eb50e"}
Nov 03 08:02:55.750044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.750015    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h" event={"ID":"8857534d-237b-4ff9-bf5b-b18cf35727f8","Type":"ContainerStarted","Data":"d7fb7cd9dac8425ba7546470660f04fd829c6873cffd8540d5e222288e5c1c62"}
Nov 03 08:02:55.750139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.750090    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h" event={"ID":"8857534d-237b-4ff9-bf5b-b18cf35727f8","Type":"ContainerStarted","Data":"8abf22f4146bfe9525349f79166f6157cef8c1a27749bb7806c3b874c5daad25"}
Nov 03 08:02:55.766505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:55.766463    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h" podStartSLOduration=2.347099251 podStartE2EDuration="2.766448922s" totalImagesPullingTime="419.349671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:55.765870115 +0000 UTC m=+525.268705628" watchObservedRunningTime="2026-11-03 08:02:55.766448922 +0000 UTC m=+525.269284435"
Nov 03 08:02:56.211744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:56.211714    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:56.755267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:56.755222    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv" event={"ID":"f0c7df50-1f48-42f9-b524-52ad120683ab","Type":"ContainerStarted","Data":"83faf4bd87194731a4dd186919ca091a9bb3757c789ca2b39a5055f730dbfe5c"}
Nov 03 08:02:56.755497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:56.755480    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:02:56.755582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:56.755529    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:02:56.785245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:56.785190    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv" podStartSLOduration=1.3387495999999999 podStartE2EDuration="1.785175629s" totalImagesPullingTime="446.426029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:56.784267546 +0000 UTC m=+526.287103065" watchObservedRunningTime="2026-11-03 08:02:56.785175629 +0000 UTC m=+526.288011143"
Nov 03 08:02:57.329730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:57.329694    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:57.436155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:57.436125    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:58.374384 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:02:58.374346    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:02:58.389783 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:58.389696    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:58.744091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:58.744033    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:02:59.482442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:59.482414    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:59.584109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:59.584079    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:02:59.695546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:02:59.695513    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:00.758467 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:00.758438    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:03:01.588045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:01.588011    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:02.766883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:02.766854    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:03:02.767307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:02.766921    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:03:03.053156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:03.053042    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:03.787089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:03.786536    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:03.901078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:03.901037    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:04.349661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:04.349625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:04.464019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:04.463993    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:04.601330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:04.601240    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:06.007928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.007891    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"]
Nov 03 08:03:06.012087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.012050    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.015161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.015142    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"default-dockercfg-lvtgz\""
Nov 03 08:03:06.018369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.018345    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"openshift-service-ca.crt\""
Nov 03 08:03:06.018589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.018575    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"kube-root-ca.crt\""
Nov 03 08:03:06.018764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.018751    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"simulator-w3-3e88b7-kserve-self-signed-certs\""
Nov 03 08:03:06.023010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.022988    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"]
Nov 03 08:03:06.026294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.026246    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-home\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.026421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.026306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tls-certs\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.026421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.026408    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-dshm\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.026762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.026471    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-model-cache\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.026762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.026597    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czb6j\" (UniqueName: \"kubernetes.io/projected/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-kube-api-access-czb6j\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.026762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.026644    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tmp-dir\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.082262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.082226    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:06.127380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127352    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-model-cache\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127429    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czb6j\" (UniqueName: \"kubernetes.io/projected/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-kube-api-access-czb6j\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127457    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tmp-dir\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127473    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-home\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127497    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tls-certs\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127521    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-dshm\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127788    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-model-cache\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127843    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tmp-dir\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.127952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.127931    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-home\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.129994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.129972    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-dshm\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.130203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.130184    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tls-certs\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.134642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.134619    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czb6j\" (UniqueName: \"kubernetes.io/projected/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-kube-api-access-czb6j\") pod \"simulator-w3-3e88b7-kserve-59455cc847-2s8nr\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") " pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.223878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.223844    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:06.326246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.326148    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:06.606214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.606136    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"]
Nov 03 08:03:06.609584 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:03:06.609555    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a0ad9a5_cbf2_4ae1_8527_ea74dfdd9938.slice/crio-c0748ceafebfed7a7b0e39880097e2387efa07fb1cd0437c6a93daa5082ae8f6 WatchSource:0}: Error finding container c0748ceafebfed7a7b0e39880097e2387efa07fb1cd0437c6a93daa5082ae8f6: Status 404 returned error can't find the container with id c0748ceafebfed7a7b0e39880097e2387efa07fb1cd0437c6a93daa5082ae8f6
Nov 03 08:03:06.806569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:06.806536    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" event={"ID":"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938","Type":"ContainerStarted","Data":"c0748ceafebfed7a7b0e39880097e2387efa07fb1cd0437c6a93daa5082ae8f6"}
Nov 03 08:03:07.432778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.432740    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:07.905121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.905042    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"]
Nov 03 08:03:07.908617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.908595    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:07.913659 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.913581    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"default-dockercfg-xt4tf\""
Nov 03 08:03:07.916959 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.916915    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"openshift-service-ca.crt\""
Nov 03 08:03:07.917115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.917003    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"kube-root-ca.crt\""
Nov 03 08:03:07.919803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.919776    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"]
Nov 03 08:03:07.938421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.938396    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"simulator-w2-6bc553-kserve-self-signed-certs\""
Nov 03 08:03:07.942951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.942930    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-home\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:07.943113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.943017    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c5eff3-19ce-4d3e-a475-0ac713d19978-tls-certs\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:07.943113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.943056    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzzlg\" (UniqueName: \"kubernetes.io/projected/50c5eff3-19ce-4d3e-a475-0ac713d19978-kube-api-access-gzzlg\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:07.943258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.943126    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-dshm\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:07.943258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.943171    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-model-cache\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:07.943258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.943188    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-tmp-dir\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:07.975103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:07.973276    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:08.044277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044246    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzzlg\" (UniqueName: \"kubernetes.io/projected/50c5eff3-19ce-4d3e-a475-0ac713d19978-kube-api-access-gzzlg\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044282    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-dshm\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044307    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-model-cache\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044416    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-tmp-dir\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044508    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-home\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044618    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c5eff3-19ce-4d3e-a475-0ac713d19978-tls-certs\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044731    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-model-cache\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044765    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-tmp-dir\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.044870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.044833    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-home\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.046680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.046659    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-dshm\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.047100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.047082    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c5eff3-19ce-4d3e-a475-0ac713d19978-tls-certs\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.055915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.055813    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzzlg\" (UniqueName: \"kubernetes.io/projected/50c5eff3-19ce-4d3e-a475-0ac713d19978-kube-api-access-gzzlg\") pod \"simulator-w2-6bc553-kserve-56b57887b5-gwnqr\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") " pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.223924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.223829    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:08.507978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.507883    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"]
Nov 03 08:03:08.511733 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:03:08.511705    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50c5eff3_19ce_4d3e_a475_0ac713d19978.slice/crio-079ed550e0af72491b336fd44492f031f85b6b4422cb352c552d060989080906 WatchSource:0}: Error finding container 079ed550e0af72491b336fd44492f031f85b6b4422cb352c552d060989080906: Status 404 returned error can't find the container with id 079ed550e0af72491b336fd44492f031f85b6b4422cb352c552d060989080906
Nov 03 08:03:08.573814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.573777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:08.818917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.818830    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" event={"ID":"50c5eff3-19ce-4d3e-a475-0ac713d19978","Type":"ContainerStarted","Data":"dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f"}
Nov 03 08:03:08.818917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.818882    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" event={"ID":"50c5eff3-19ce-4d3e-a475-0ac713d19978","Type":"ContainerStarted","Data":"079ed550e0af72491b336fd44492f031f85b6b4422cb352c552d060989080906"}
Nov 03 08:03:08.820558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.820530    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" event={"ID":"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938","Type":"ContainerStarted","Data":"b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298"}
Nov 03 08:03:08.833110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.833039    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podStartSLOduration=1.833022374 podStartE2EDuration="1.833022374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:08.832399225 +0000 UTC m=+538.335234750" watchObservedRunningTime="2026-11-03 08:03:08.833022374 +0000 UTC m=+538.335857888"
Nov 03 08:03:08.847425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.847373    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podStartSLOduration=2.6490412340000002 podStartE2EDuration="3.847361617s" totalImagesPullingTime="1.198320383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:08.846786544 +0000 UTC m=+538.349622060" watchObservedRunningTime="2026-11-03 08:03:08.847361617 +0000 UTC m=+538.350197131"
Nov 03 08:03:08.932114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:08.932053    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:09.042016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:09.041976    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:09.224210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:09.224166    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:09.327256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:09.327223    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:09.444931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:09.444900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:09.579195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:09.579118    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:09.722267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:09.722231    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:09.972932 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:09.972905    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:10.104250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:10.104223    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:10.225996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:10.225920    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:10.803402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:10.803362    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:11.502137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:11.502100    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:11.934851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:11.934815    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:12.285695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:12.285617    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:12.738239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:12.738199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:12.981626 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:12.981593    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:13.507309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.507275    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:13.859644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.859556    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"]
Nov 03 08:03:13.863569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.863538    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:13.872964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.872933    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"kube-root-ca.crt\""
Nov 03 08:03:13.872964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.872937    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"openshift-service-ca.crt\""
Nov 03 08:03:13.873521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.873499    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"simulator-w0-76ef13-kserve-self-signed-certs\""
Nov 03 08:03:13.873651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.873548    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"default-dockercfg-fxzvz\""
Nov 03 08:03:13.875649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.875616    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"]
Nov 03 08:03:13.902875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.902837    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-home\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:13.903107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.902917    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-dshm\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:13.903107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.903026    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-model-cache\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:13.903107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.903100    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-tmp-dir\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:13.903293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.903155    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxs5\" (UniqueName: \"kubernetes.io/projected/35388dcd-6433-4f50-a084-947740876acb-kube-api-access-8rxs5\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:13.903293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.903250    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35388dcd-6433-4f50-a084-947740876acb-tls-certs\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:13.935376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:13.935342    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:14.004437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.004387    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-dshm\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.004437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.004443    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-model-cache\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.004987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.004488    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-tmp-dir\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.004987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.004520    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxs5\" (UniqueName: \"kubernetes.io/projected/35388dcd-6433-4f50-a084-947740876acb-kube-api-access-8rxs5\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.004987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.004668    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35388dcd-6433-4f50-a084-947740876acb-tls-certs\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.004987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.004748    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-home\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.005208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.005103    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-home\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.005208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.005173    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-tmp-dir\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.005301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.005214    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-model-cache\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.007418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.007388    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-dshm\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.007574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.007478    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35388dcd-6433-4f50-a084-947740876acb-tls-certs\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.011470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.011447    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxs5\" (UniqueName: \"kubernetes.io/projected/35388dcd-6433-4f50-a084-947740876acb-kube-api-access-8rxs5\") pod \"simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") " pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.054873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.054840    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:14.179901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.179808    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:14.182027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.181996    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:14.311270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.311237    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:14.485413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.485293    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"]
Nov 03 08:03:14.490218 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:03:14.490188    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35388dcd_6433_4f50_a084_947740876acb.slice/crio-d5ea27405540c670c117bf8eba4df9271f124b2fb02fd4be783d3c6214c8327c WatchSource:0}: Error finding container d5ea27405540c670c117bf8eba4df9271f124b2fb02fd4be783d3c6214c8327c: Status 404 returned error can't find the container with id d5ea27405540c670c117bf8eba4df9271f124b2fb02fd4be783d3c6214c8327c
Nov 03 08:03:14.588859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.588827    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:14.850689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.850607    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" event={"ID":"35388dcd-6433-4f50-a084-947740876acb","Type":"ContainerStarted","Data":"9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a"}
Nov 03 08:03:14.850689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.850657    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" event={"ID":"35388dcd-6433-4f50-a084-947740876acb","Type":"ContainerStarted","Data":"d5ea27405540c670c117bf8eba4df9271f124b2fb02fd4be783d3c6214c8327c"}
Nov 03 08:03:14.868459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:14.868400    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podStartSLOduration=1.868380579 podStartE2EDuration="1.868380579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:14.866204439 +0000 UTC m=+544.369039956" watchObservedRunningTime="2026-11-03 08:03:14.868380579 +0000 UTC m=+544.371216096"
Nov 03 08:03:15.081035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:15.081002    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:15.180976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:15.180955    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:15.211884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:15.211846    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:15.341522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:15.341490    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:15.934475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:15.934442    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:16.326939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:16.326861    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:16.339368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:16.339343    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:16.870255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:16.870229    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:03:17.624690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:17.624656    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:17.840986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:17.840953    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:18.180548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:18.180516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:18.224322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:18.224295    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:18.237175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:18.237152    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:18.290556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:18.290532    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:18.881262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:18.881237    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:03:19.139525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:19.139445    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:20.844576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:20.844533    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:21.824925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.824890    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"]
Nov 03 08:03:21.828712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.828689    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.833779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.833755    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"kube-root-ca.crt\""
Nov 03 08:03:21.833930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.833755    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"openshift-service-ca.crt\""
Nov 03 08:03:21.833991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.833938    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"default-dockercfg-9w92k\""
Nov 03 08:03:21.834043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.833996    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"simulator-w6-bd085d-kserve-self-signed-certs\""
Nov 03 08:03:21.840497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.840473    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"]
Nov 03 08:03:21.875245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.875206    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:21.892424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.892400    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-tmp-dir\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.892595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.892434    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-model-cache\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.892595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.892504    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abdfcc3-5815-49a3-8459-6324e5013873-tls-certs\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.892719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.892694    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrft\" (UniqueName: \"kubernetes.io/projected/2abdfcc3-5815-49a3-8459-6324e5013873-kube-api-access-gbrft\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.892786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.892755    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-dshm\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.892842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.892817    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-home\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.993443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.993410    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abdfcc3-5815-49a3-8459-6324e5013873-tls-certs\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.993661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.993478    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrft\" (UniqueName: \"kubernetes.io/projected/2abdfcc3-5815-49a3-8459-6324e5013873-kube-api-access-gbrft\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.993661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.993525    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-dshm\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.993920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.993893    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-home\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.994006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.993961    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-tmp-dir\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.994006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.993998    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-model-cache\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.994239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.994215    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-home\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.994459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.994356    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-model-cache\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.994534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.994475    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-tmp-dir\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:21.996387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:21.996287    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-dshm\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:22.003096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.000653    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:22.004231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.004206    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrft\" (UniqueName: \"kubernetes.io/projected/2abdfcc3-5815-49a3-8459-6324e5013873-kube-api-access-gbrft\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:22.006495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.006475    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abdfcc3-5815-49a3-8459-6324e5013873-tls-certs\") pod \"simulator-w6-bd085d-kserve-64564f968-56454\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") " pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:22.144188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.144096    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:22.430782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.430708    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"]
Nov 03 08:03:22.434093 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:03:22.434040    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2abdfcc3_5815_49a3_8459_6324e5013873.slice/crio-af13c4464bdbd3f1972e5009fa2123c513e7635b1acca4e69f2fd4a5e49610be WatchSource:0}: Error finding container af13c4464bdbd3f1972e5009fa2123c513e7635b1acca4e69f2fd4a5e49610be: Status 404 returned error can't find the container with id af13c4464bdbd3f1972e5009fa2123c513e7635b1acca4e69f2fd4a5e49610be
Nov 03 08:03:22.769336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.769301    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:22.903410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.903374    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" event={"ID":"2abdfcc3-5815-49a3-8459-6324e5013873","Type":"ContainerStarted","Data":"b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c"}
Nov 03 08:03:22.903869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.903420    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" event={"ID":"2abdfcc3-5815-49a3-8459-6324e5013873","Type":"ContainerStarted","Data":"af13c4464bdbd3f1972e5009fa2123c513e7635b1acca4e69f2fd4a5e49610be"}
Nov 03 08:03:22.918867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:22.918790    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podStartSLOduration=1.918773821 podStartE2EDuration="1.918773821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:22.91743242 +0000 UTC m=+552.420267934" watchObservedRunningTime="2026-11-03 08:03:22.918773821 +0000 UTC m=+552.421609337"
Nov 03 08:03:23.144822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:23.144738    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:23.461200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:23.461144    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:24.036501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:24.036432    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:24.180549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:24.180521    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:24.193131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:24.193103    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:24.274098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:24.274043    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:24.925404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:24.925377    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:03:25.052857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:25.052824    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:26.042715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:26.042680    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:26.745290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:26.745259    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:27.441574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:27.441539    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:28.145609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:28.145574    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:28.375834 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:03:28.375801    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:03:28.730828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:28.730795    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:28.925865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:28.925838    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:29.639010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:29.638980    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:30.341780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:30.341739    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:31.151365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:31.151332    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:32.023078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:32.023030    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:32.144725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:32.144685    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:32.157299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:32.157274    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:32.767920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:32.767896    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:32.963043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:32.963011    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:03:33.655966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:33.655928    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:34.458905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:34.458845    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:35.149577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:35.149548    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:35.257559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:35.257526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:36.046680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:36.046650    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:37.021159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:37.021119    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:37.862401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:37.862333    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:38.505671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:38.505637    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:38.619738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:38.619702    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:39.265126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.265096    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:39.499567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.499403    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"]
Nov 03 08:03:39.506301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.506275    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.514900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.514880    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"premium-w2-6bc553-kserve-self-signed-certs\""
Nov 03 08:03:39.518641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.518588    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"]
Nov 03 08:03:39.561702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.561670    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:39.581656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.581623    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tls-certs\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.581814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.581674    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flgx\" (UniqueName: \"kubernetes.io/projected/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-kube-api-access-9flgx\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.581865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.581816    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-home\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.581916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.581879    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-dshm\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.581954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.581936    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-model-cache\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.582107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.582086    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tmp-dir\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.685823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.685758    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-dshm\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.686529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.686506    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-model-cache\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.691236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.690595    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-model-cache\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.691428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.691405    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tmp-dir\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.694960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.691627    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tmp-dir\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.694960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.691629    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tls-certs\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.694960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.691707    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flgx\" (UniqueName: \"kubernetes.io/projected/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-kube-api-access-9flgx\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.694960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.691745    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-home\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.694960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.692104    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-home\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.700568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.700471    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tls-certs\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.700736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.700640    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flgx\" (UniqueName: \"kubernetes.io/projected/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-kube-api-access-9flgx\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.704185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.704158    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-dshm\") pod \"premium-w2-6bc553-kserve-7f684958bd-9pnx6\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") " pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:39.820329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:39.820263    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:40.047306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:40.047264    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:40.104493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:40.104404    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"]
Nov 03 08:03:40.108218 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:03:40.108191    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb08503f1_9bb1_4a0f_b1ab_09f6e8e21838.slice/crio-d4bb5ada08488cc3cb4988620fc5d2365cc591d7b1ead1ace18146776a8ff4cb WatchSource:0}: Error finding container d4bb5ada08488cc3cb4988620fc5d2365cc591d7b1ead1ace18146776a8ff4cb: Status 404 returned error can't find the container with id d4bb5ada08488cc3cb4988620fc5d2365cc591d7b1ead1ace18146776a8ff4cb
Nov 03 08:03:40.895163 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:40.895134    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:40.989334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:40.989286    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" event={"ID":"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838","Type":"ContainerStarted","Data":"bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017"}
Nov 03 08:03:40.989334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:40.989337    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" event={"ID":"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838","Type":"ContainerStarted","Data":"d4bb5ada08488cc3cb4988620fc5d2365cc591d7b1ead1ace18146776a8ff4cb"}
Nov 03 08:03:41.003509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:41.003455    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podStartSLOduration=2.003439069 podStartE2EDuration="2.003439069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:41.002573211 +0000 UTC m=+570.505408725" watchObservedRunningTime="2026-11-03 08:03:41.003439069 +0000 UTC m=+570.506274579"
Nov 03 08:03:41.017315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:41.017287    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:41.820968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:41.820924    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:41.870878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:41.870846    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:42.739437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:42.739407    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:43.550635 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:43.550604    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:44.354109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:44.354053    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:45.053132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:45.053095    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:45.850020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:45.849987    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:46.543141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:46.543111    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:47.261024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:47.260989    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:48.045425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:48.045369    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:48.744946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:48.744917    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:49.599265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:49.599230    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:49.821106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:49.821081    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:49.833961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:49.833933    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:50.042629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.042600    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:03:50.576660 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.576630    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"]
Nov 03 08:03:50.581117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.581073    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.589245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.589189    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"premium-w6-bd085d-kserve-self-signed-certs\""
Nov 03 08:03:50.591841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.591816    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"]
Nov 03 08:03:50.597815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.597790    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:50.602112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.602087    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-dshm\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.602228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.602146    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-model-cache\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.602301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.602233    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrvz\" (UniqueName: \"kubernetes.io/projected/ca64b314-7012-47d5-9e3e-361468b2b02e-kube-api-access-lkrvz\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.602301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.602273    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca64b314-7012-47d5-9e3e-361468b2b02e-tls-certs\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.602425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.602399    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-tmp-dir\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.602603 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.602583    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-home\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.704240 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.704199    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-model-cache\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.704980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.704946    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrvz\" (UniqueName: \"kubernetes.io/projected/ca64b314-7012-47d5-9e3e-361468b2b02e-kube-api-access-lkrvz\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.705358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.705337    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca64b314-7012-47d5-9e3e-361468b2b02e-tls-certs\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.705673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.705655    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-tmp-dir\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.707850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.707831    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-home\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.708104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.704867    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-model-cache\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.708611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.708550    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-home\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.709856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.708839    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-dshm\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.710056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.707764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-tmp-dir\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.739405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.728791    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:50.739405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.729569    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrvz\" (UniqueName: \"kubernetes.io/projected/ca64b314-7012-47d5-9e3e-361468b2b02e-kube-api-access-lkrvz\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.739405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.737713    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca64b314-7012-47d5-9e3e-361468b2b02e-tls-certs\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.739405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.738653    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-dshm\") pod \"premium-w6-bd085d-kserve-5d4f747589-dhd69\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") " pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:50.894404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:50.894321    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:51.180107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:51.179979    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"]
Nov 03 08:03:51.183968 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:03:51.183942    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca64b314_7012_47d5_9e3e_361468b2b02e.slice/crio-7abcd3fd61948deedb41efa0f1a65301e53553feb7568804c5b2e7271c6733d7 WatchSource:0}: Error finding container 7abcd3fd61948deedb41efa0f1a65301e53553feb7568804c5b2e7271c6733d7: Status 404 returned error can't find the container with id 7abcd3fd61948deedb41efa0f1a65301e53553feb7568804c5b2e7271c6733d7
Nov 03 08:03:51.533514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:51.533477    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:52.043817 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:52.043783    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" event={"ID":"ca64b314-7012-47d5-9e3e-361468b2b02e","Type":"ContainerStarted","Data":"621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941"}
Nov 03 08:03:52.044265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:52.043823    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" event={"ID":"ca64b314-7012-47d5-9e3e-361468b2b02e","Type":"ContainerStarted","Data":"7abcd3fd61948deedb41efa0f1a65301e53553feb7568804c5b2e7271c6733d7"}
Nov 03 08:03:52.057731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:52.057677    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podStartSLOduration=2.057644978 podStartE2EDuration="2.057644978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:52.05687266 +0000 UTC m=+581.559708195" watchObservedRunningTime="2026-11-03 08:03:52.057644978 +0000 UTC m=+581.560480492"
Nov 03 08:03:52.251588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:52.251554    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:52.894843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:52.894789    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:03:53.155305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:53.155228    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:53.971540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:53.971508    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:54.762981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:54.762946    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:55.467385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:55.467330    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:55.592088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:55.588164    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:56.271118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:56.271088    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:57.067005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:57.066973    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:57.844150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:57.844105    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:58.376202 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:03:58.376166    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:03:59.125146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:59.125111    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:03:59.849671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:03:59.849620    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:00.543180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:00.543135    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:00.894793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:00.894704    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:04:00.907454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:00.907432    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:04:01.096966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:01.096938    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:04:01.254448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:01.254411    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:02.039968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:02.039933    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:02.939923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:02.939891    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:03.713771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:03.713709    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:04.445028 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:04.444989    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:05.185055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:05.185020    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:05.855414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:05.855383    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:06.543550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:06.543514    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:07.245241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:07.245208    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:07.935588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:07.935556    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:08.756183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:08.756143    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:09.766513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:09.766481    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:10.665416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:10.665381    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:11.539614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:11.539578    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:11.647317 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:11.647287    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:12.519794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:12.519755    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:13.615375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:13.615339    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:14.344149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:14.344082    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:15.250678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.250573    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:15.502211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.502138    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"]
Nov 03 08:04:15.507043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.507023    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.512617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.512597    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6bc553-openshift-default-dockercfg-fcnbk\""
Nov 03 08:04:15.517603 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.517585    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"]
Nov 03 08:04:15.669176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669141    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669186    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669212    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a6364-3e36-4b74-a399-286cccbb26d5-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669350    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669380    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669467 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669417    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669467 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669451    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669488    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a6364-3e36-4b74-a399-286cccbb26d5-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.669530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.669504    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh72n\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-kube-api-access-mh72n\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.770753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770683    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.770753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770720    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.770976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.770976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770788    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.770976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770823    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a6364-3e36-4b74-a399-286cccbb26d5-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.770976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770867    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh72n\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-kube-api-access-mh72n\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.770976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.771271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.770981    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.771271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.771016    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a6364-3e36-4b74-a399-286cccbb26d5-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.771271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.771172    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.771271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.771227    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.771474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.771389    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.771474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.771439    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.771599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.771579    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a6364-3e36-4b74-a399-286cccbb26d5-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.773269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.773245    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.773545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.773526    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a6364-3e36-4b74-a399-286cccbb26d5-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.784446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.784421    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.784885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.784863    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh72n\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-kube-api-access-mh72n\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:15.820643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:15.820620    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:16.090129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.090034    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:16.098052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.098027    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"]
Nov 03 08:04:16.101138 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:04:16.101112    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921a6364_3e36_4b74_a399_286cccbb26d5.slice/crio-c2e9f2b5ecb0c673015be96977aa770879acc3a83cd4174f1490c72d7475d16b WatchSource:0}: Error finding container c2e9f2b5ecb0c673015be96977aa770879acc3a83cd4174f1490c72d7475d16b: Status 404 returned error can't find the container with id c2e9f2b5ecb0c673015be96977aa770879acc3a83cd4174f1490c72d7475d16b
Nov 03 08:04:16.104558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.104467    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:16.104812 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.104791    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:16.105012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.104972    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:16.154517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.154490    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" event={"ID":"921a6364-3e36-4b74-a399-286cccbb26d5","Type":"ContainerStarted","Data":"c2e9f2b5ecb0c673015be96977aa770879acc3a83cd4174f1490c72d7475d16b"}
Nov 03 08:04:16.740594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.740554    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:16.863128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:16.863093    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:17.160778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:17.160689    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" event={"ID":"921a6364-3e36-4b74-a399-286cccbb26d5","Type":"ContainerStarted","Data":"0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca"}
Nov 03 08:04:17.179043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:17.178992    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podStartSLOduration=2.178978633 podStartE2EDuration="2.178978633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:17.177207431 +0000 UTC m=+606.680042924" watchObservedRunningTime="2026-11-03 08:04:17.178978633 +0000 UTC m=+606.681814200"
Nov 03 08:04:17.628265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:17.628235    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:17.822583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:17.822537    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:17.824122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:17.824097    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 08:04:17.824257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:17.824149    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 08:04:18.344262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:18.344226    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:18.822089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:18.821500    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 08:04:18.822089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:18.821577    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 08:04:19.144765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:19.144659    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:19.822482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:19.822447    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 08:04:19.822688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:19.822522    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 08:04:19.836775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:19.836751    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:20.540229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:20.540188    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:20.822590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:20.822489    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 08:04:20.822590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:20.822557    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 08:04:21.245384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:21.245345    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:21.822091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:21.821551    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 08:04:21.822091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:21.821616    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 08:04:22.041750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:22.041714    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:22.826538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:22.826509    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:22.827018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:22.826905    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:22.827590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:22.827567    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:04:22.854974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:22.854949    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:23.757918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:23.757875    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:24.660714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:24.660671    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:25.460947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:25.460907    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:26.265491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:26.265463    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:27.110695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:27.110665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:27.937355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:27.937322    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:28.375547 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:04:28.375466    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:04:28.644989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:28.644889    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:29.454879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:29.454842    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:30.191605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.191548    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:30.662027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.661954    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"]
Nov 03 08:04:30.665405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.665385    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.677534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.677511    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"]
Nov 03 08:04:30.711509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.711478    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-19bb0a-openshift-default-dockercfg-2phtd\""
Nov 03 08:04:30.756580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756552    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.756580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756588    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.756771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756620    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.756771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756680    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.756837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756772    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwt4h\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-kube-api-access-fwt4h\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.756837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756822    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.756902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756843    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.756902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756887    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad54e90-0786-4e0b-add5-66205e0ca62b-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.757017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.756997    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858442    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858514    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad54e90-0786-4e0b-add5-66205e0ca62b-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858576    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858644    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858678    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858725    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858748    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858780    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwt4h\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-kube-api-access-fwt4h\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.858968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.858813    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.859044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.859019    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.859150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.859101    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.859213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.859164    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.859213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.859182    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.859542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.859491    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad54e90-0786-4e0b-add5-66205e0ca62b-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.861640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.861618    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.862174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.862153    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.867435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.867410    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwt4h\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-kube-api-access-fwt4h\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.867657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.867637    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:30.980195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:30.980170    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:31.099752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:31.099724    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:31.260045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:31.259965    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"]
Nov 03 08:04:31.263409 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:04:31.263380    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad54e90_0786_4e0b_add5_66205e0ca62b.slice/crio-06a91ae194141c649e07fc81ce305ce5d002e7f95263ddadd34b07cf64a2e0d9 WatchSource:0}: Error finding container 06a91ae194141c649e07fc81ce305ce5d002e7f95263ddadd34b07cf64a2e0d9: Status 404 returned error can't find the container with id 06a91ae194141c649e07fc81ce305ce5d002e7f95263ddadd34b07cf64a2e0d9
Nov 03 08:04:31.265648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:31.265619    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:31.265712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:31.265677    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:31.265712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:31.265705    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:31.966942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:31.966823    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:32.254258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:32.254182    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" event={"ID":"9ad54e90-0786-4e0b-add5-66205e0ca62b","Type":"ContainerStarted","Data":"eb03b801d5648ea65e9761ba535d41ceafad033305af9c3e8cc72f243a9ee176"}
Nov 03 08:04:32.254258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:32.254233    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" event={"ID":"9ad54e90-0786-4e0b-add5-66205e0ca62b","Type":"ContainerStarted","Data":"06a91ae194141c649e07fc81ce305ce5d002e7f95263ddadd34b07cf64a2e0d9"}
Nov 03 08:04:32.272116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:32.272038    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podStartSLOduration=2.272024471 podStartE2EDuration="2.272024471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:32.27152024 +0000 UTC m=+621.774355755" watchObservedRunningTime="2026-11-03 08:04:32.272024471 +0000 UTC m=+621.774859986"
Nov 03 08:04:32.867417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:32.867378    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:32.980581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:32.980545    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:32.982399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:32.982371    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 08:04:32.982524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:32.982420    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 08:04:33.980699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:33.980665    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 08:04:33.981195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:33.980716    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 08:04:34.315033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:34.314918    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:34.980813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:34.980782    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 08:04:34.981225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:34.980838    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 08:04:35.268994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:35.268913    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:35.981028 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:35.980991    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 08:04:35.981504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:35.981040    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 08:04:36.263134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:36.263023    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:36.981370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:36.981342    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 08:04:36.981743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:36.981404    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 08:04:37.175006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:37.174975    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:38.160364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:38.160333    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:38.161139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:38.161109    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:38.161967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:38.161947    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:04:38.221185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:38.221144    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:39.304277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:39.304248    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:40.075955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:40.075928    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:40.827485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:40.827445    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:41.742843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:41.742811    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:42.450563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:42.450530    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:43.307957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:43.307922    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:44.053864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:44.053831    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:44.621581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:44.621543    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:44.730892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:44.730852    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:44.844211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:44.844167    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:45.497529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.497500    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"]
Nov 03 08:04:45.501519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.501496    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.509837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.509801    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e88b7-openshift-default-dockercfg-k6k9p\""
Nov 03 08:04:45.515642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.515618    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"]
Nov 03 08:04:45.609083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.609038    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:45.653748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.653712    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e482cc-190c-4a3f-aa6f-59ff75972735-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.653748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.653750    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.654014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.653784    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.654014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.653948    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.654161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.654032    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxclw\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-kube-api-access-bxclw\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.654161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.654090    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.654161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.654149    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.654288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.654190    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.654288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.654246    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.754793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.754702    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.754793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.754743    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxclw\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-kube-api-access-bxclw\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.754793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.754772    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.754828    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.754971    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755025    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755186    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755229    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e482cc-190c-4a3f-aa6f-59ff75972735-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755237    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755251    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755270    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755295    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755597    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.755885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.755866    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e482cc-190c-4a3f-aa6f-59ff75972735-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.757438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.757417    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.757765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.757748    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.761669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.761639    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.761886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.761870    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxclw\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-kube-api-access-bxclw\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:45.815738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:45.815700    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:46.094175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.094097    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"]
Nov 03 08:04:46.097546 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:04:46.097523    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e482cc_190c_4a3f_aa6f_59ff75972735.slice/crio-455b5aa215037dd3466e80c42b4e6abe46e0e47c8afbee0668cd5fb05bd2bc07 WatchSource:0}: Error finding container 455b5aa215037dd3466e80c42b4e6abe46e0e47c8afbee0668cd5fb05bd2bc07: Status 404 returned error can't find the container with id 455b5aa215037dd3466e80c42b4e6abe46e0e47c8afbee0668cd5fb05bd2bc07
Nov 03 08:04:46.099959 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.099920    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:46.100050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.099981    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:46.100050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.100011    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:04:46.326272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.326232    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" event={"ID":"05e482cc-190c-4a3f-aa6f-59ff75972735","Type":"ContainerStarted","Data":"371c029c2651a6c9d9dff6b049744c4e36ef7885f0cb291241a3502a31a6c1c4"}
Nov 03 08:04:46.326439 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.326298    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" event={"ID":"05e482cc-190c-4a3f-aa6f-59ff75972735","Type":"ContainerStarted","Data":"455b5aa215037dd3466e80c42b4e6abe46e0e47c8afbee0668cd5fb05bd2bc07"}
Nov 03 08:04:46.344589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.344509    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podStartSLOduration=1.344497146 podStartE2EDuration="1.344497146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:46.343025962 +0000 UTC m=+635.845861477" watchObservedRunningTime="2026-11-03 08:04:46.344497146 +0000 UTC m=+635.847332658"
Nov 03 08:04:46.532427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.532382    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:46.819197 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:46.819155    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:47.346038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:47.345999    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:47.826238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:47.826200    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 08:04:47.826777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:47.826262    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 08:04:48.056074 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:48.056026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:48.818089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:48.816496    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 08:04:48.818089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:48.816561    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 08:04:48.982993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:48.982957    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:49.751420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:49.751342    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:49.820087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:49.819476    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 08:04:49.820087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:49.819552    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 08:04:50.642884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:50.642855    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:50.818416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:50.818382    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 08:04:50.818688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:50.818661    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 08:04:51.515612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:51.515577    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:51.818142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:51.817443    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 08:04:51.818142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:51.817504    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 08:04:52.241838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:52.241802    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:53.249149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:53.249096    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:53.629281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:53.629198    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:53.629501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:53.629482    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:53.630203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:53.630183    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:04:54.068544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:54.068516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:54.866431 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:54.866398    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:55.843454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:55.843425    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:56.014134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.014100    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"]
Nov 03 08:04:56.017560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.017544    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.020208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.020187    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:04:56.020301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.020287    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:04:56.025918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.025883    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"]
Nov 03 08:04:56.101548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.101476    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgc4z\" (UniqueName: \"kubernetes.io/projected/40dc4f74-a658-488e-9c02-a08c20a664bb-kube-api-access-mgc4z\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.101548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.101531    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.101699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.101654    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.203137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.203110    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgc4z\" (UniqueName: \"kubernetes.io/projected/40dc4f74-a658-488e-9c02-a08c20a664bb-kube-api-access-mgc4z\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.203287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.203154    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.203287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.203203    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.203844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.203825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.205578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.205560    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.210498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.210479    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgc4z\" (UniqueName: \"kubernetes.io/projected/40dc4f74-a658-488e-9c02-a08c20a664bb-kube-api-access-mgc4z\") pod \"maas-api-revoke-sub-fe6a9d25-ltjdr\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.332199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.332168    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:04:56.606075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.606017    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"]
Nov 03 08:04:56.610054 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:04:56.610022    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dc4f74_a658_488e_9c02_a08c20a664bb.slice/crio-dcf9c7f85292ae294587ee99b2462d4ffff5daf8aab9658bafce318d764f85fb WatchSource:0}: Error finding container dcf9c7f85292ae294587ee99b2462d4ffff5daf8aab9658bafce318d764f85fb: Status 404 returned error can't find the container with id dcf9c7f85292ae294587ee99b2462d4ffff5daf8aab9658bafce318d764f85fb
Nov 03 08:04:56.730825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:56.730795    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:57.407641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:57.407608    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr" event={"ID":"40dc4f74-a658-488e-9c02-a08c20a664bb","Type":"ContainerStarted","Data":"dcf9c7f85292ae294587ee99b2462d4ffff5daf8aab9658bafce318d764f85fb"}
Nov 03 08:04:57.486963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:57.486929    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:58.340512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:58.340453    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:58.398087 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:04:58.397972    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:04:59.147232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:59.147199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:04:59.841631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:04:59.841598    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:00.424071 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:00.424034    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr" event={"ID":"40dc4f74-a658-488e-9c02-a08c20a664bb","Type":"ContainerStarted","Data":"e8bdc8ed5469d2e67714d56727bff35da1f95a4d48cf776b0b095a4bae1df89d"}
Nov 03 08:05:00.438403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:00.438341    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr" podStartSLOduration=2.084267841 podStartE2EDuration="5.438323852s" totalImagesPullingTime="3.354056011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:00.437126943 +0000 UTC m=+649.939962457" watchObservedRunningTime="2026-11-03 08:05:00.438323852 +0000 UTC m=+649.941159367"
Nov 03 08:05:00.686433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:00.686406    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:01.451739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:01.451702    6430 generic.go:356] "Generic (PLEG): container finished" podID="40dc4f74-a658-488e-9c02-a08c20a664bb" containerID="e8bdc8ed5469d2e67714d56727bff35da1f95a4d48cf776b0b095a4bae1df89d" exitCode=0
Nov 03 08:05:01.452874 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:01.452338    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr" event={"ID":"40dc4f74-a658-488e-9c02-a08c20a664bb","Type":"ContainerDied","Data":"e8bdc8ed5469d2e67714d56727bff35da1f95a4d48cf776b0b095a4bae1df89d"}
Nov 03 08:05:02.353352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.353313    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:02.685512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.685492    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:05:02.789477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.788928    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-openshift-ca\") pod \"40dc4f74-a658-488e-9c02-a08c20a664bb\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") "
Nov 03 08:05:02.789477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.788995    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dc4f74-a658-488e-9c02-a08c20a664bb-kube-api-access-mgc4z\" (UniqueName: \"kubernetes.io/projected/40dc4f74-a658-488e-9c02-a08c20a664bb-kube-api-access-mgc4z\") pod \"40dc4f74-a658-488e-9c02-a08c20a664bb\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") "
Nov 03 08:05:02.789477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.789044    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-xks-ca\") pod \"40dc4f74-a658-488e-9c02-a08c20a664bb\" (UID: \"40dc4f74-a658-488e-9c02-a08c20a664bb\") "
Nov 03 08:05:02.789477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.789279    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-openshift-ca" pod "40dc4f74-a658-488e-9c02-a08c20a664bb" (UID: "40dc4f74-a658-488e-9c02-a08c20a664bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:02.789477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.789451    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:02.791403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.791372    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-xks-ca" pod "40dc4f74-a658-488e-9c02-a08c20a664bb" (UID: "40dc4f74-a658-488e-9c02-a08c20a664bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:02.791533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.791497    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dc4f74-a658-488e-9c02-a08c20a664bb-kube-api-access-mgc4z" pod "40dc4f74-a658-488e-9c02-a08c20a664bb" (UID: "40dc4f74-a658-488e-9c02-a08c20a664bb"). InnerVolumeSpecName "kube-api-access-mgc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:02.890080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.890026    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgc4z\" (UniqueName: \"kubernetes.io/projected/40dc4f74-a658-488e-9c02-a08c20a664bb-kube-api-access-mgc4z\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:02.890080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:02.890052    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40dc4f74-a658-488e-9c02-a08c20a664bb-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:03.468384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:03.468340    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr" event={"ID":"40dc4f74-a658-488e-9c02-a08c20a664bb","Type":"ContainerDied","Data":"dcf9c7f85292ae294587ee99b2462d4ffff5daf8aab9658bafce318d764f85fb"}
Nov 03 08:05:03.468550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:03.468407    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcf9c7f85292ae294587ee99b2462d4ffff5daf8aab9658bafce318d764f85fb"
Nov 03 08:05:03.468550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:03.468427    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"
Nov 03 08:05:04.367705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:04.367675    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:05.386352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:05.386301    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:05.654200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:05.654117    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:06.503794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:06.503768    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:07.352397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:07.352369    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:08.242356 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:08.242325    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:09.150159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:09.150120    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:09.939852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:09.939826    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:10.745951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:10.745891    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:11.440900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:11.440868    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:11.586350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:11.586321    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:11.719811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:11.719721    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:12.335650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:12.335616    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:13.134567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.134529    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:13.690529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.690497    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"]
Nov 03 08:05:13.690947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.690933    6430 cpu_manager.go:414] "containerMap: removing container" podUID="40dc4f74-a658-488e-9c02-a08c20a664bb" containerName="revoke-keys"
Nov 03 08:05:13.690990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.690952    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40dc4f74-a658-488e-9c02-a08c20a664bb" containerName="revoke-keys"
Nov 03 08:05:13.691024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.691018    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="40dc4f74-a658-488e-9c02-a08c20a664bb" containerName="revoke-keys"
Nov 03 08:05:13.695534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.695502    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.713178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.713139    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"kube-root-ca.crt\""
Nov 03 08:05:13.713178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.713147    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"openshift-service-ca.crt\""
Nov 03 08:05:13.713734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.713713    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"default-dockercfg-vkqgz\""
Nov 03 08:05:13.714327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.714308    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"]
Nov 03 08:05:13.717162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.717136    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"distinct-w5-ffe7a8-kserve-self-signed-certs\""
Nov 03 08:05:13.834088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.831359    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:13.838086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.836122    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4194f3d-d12c-417b-a715-7fedf4dfd732-tls-certs\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.838086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.836255    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-dshm\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.838086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.836286    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-tmp-dir\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.838086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.836394    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fnts\" (UniqueName: \"kubernetes.io/projected/d4194f3d-d12c-417b-a715-7fedf4dfd732-kube-api-access-8fnts\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.838086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.836489    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-model-cache\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.838086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.836519    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-home\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.937650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.937615    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fnts\" (UniqueName: \"kubernetes.io/projected/d4194f3d-d12c-417b-a715-7fedf4dfd732-kube-api-access-8fnts\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.937879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.937712    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-model-cache\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.937879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.937746    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-home\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.937879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.937782    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4194f3d-d12c-417b-a715-7fedf4dfd732-tls-certs\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.937879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.937817    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-dshm\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.937879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.937842    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-tmp-dir\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.938232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.938179    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-model-cache\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.938362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.938214    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-home\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.938429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.938366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-tmp-dir\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.940692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.940631    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-dshm\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.940934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.940914    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4194f3d-d12c-417b-a715-7fedf4dfd732-tls-certs\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.945786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.945753    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fnts\" (UniqueName: \"kubernetes.io/projected/d4194f3d-d12c-417b-a715-7fedf4dfd732-kube-api-access-8fnts\") pod \"distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:13.962282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:13.962196    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:14.010589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:14.010307    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:14.307994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:14.307908    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"]
Nov 03 08:05:14.311940 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:05:14.311912    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4194f3d_d12c_417b_a715_7fedf4dfd732.slice/crio-a9d613d679f12784c1b359ca7ee6409b89331fd15a366e318fc5a00a080e5acd WatchSource:0}: Error finding container a9d613d679f12784c1b359ca7ee6409b89331fd15a366e318fc5a00a080e5acd: Status 404 returned error can't find the container with id a9d613d679f12784c1b359ca7ee6409b89331fd15a366e318fc5a00a080e5acd
Nov 03 08:05:14.533926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:14.533891    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" event={"ID":"d4194f3d-d12c-417b-a715-7fedf4dfd732","Type":"ContainerStarted","Data":"2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad"}
Nov 03 08:05:14.533926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:14.533931    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" event={"ID":"d4194f3d-d12c-417b-a715-7fedf4dfd732","Type":"ContainerStarted","Data":"a9d613d679f12784c1b359ca7ee6409b89331fd15a366e318fc5a00a080e5acd"}
Nov 03 08:05:14.554979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:14.554923    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podStartSLOduration=1.554903758 podStartE2EDuration="1.554903758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:14.552869654 +0000 UTC m=+664.055705170" watchObservedRunningTime="2026-11-03 08:05:14.554903758 +0000 UTC m=+664.057739273"
Nov 03 08:05:14.866913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:14.866880    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:15.011838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.011801    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:15.314884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.314845    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"]
Nov 03 08:05:15.322244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.322214    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.329122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.328842    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-76ef13-openshift-default-dockercfg-4ftn9\""
Nov 03 08:05:15.339623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.339592    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"]
Nov 03 08:05:15.457733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.457693    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.457994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.457970    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.458185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.458164    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnpbl\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-kube-api-access-gnpbl\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.458340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.458323    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.458465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.458450    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.458597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.458583    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f954b829-91a3-4aee-925c-363f1cfa9824-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.458717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.458702    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f954b829-91a3-4aee-925c-363f1cfa9824-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.458832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.458818    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.459004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.458989    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.564989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.564867    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.565673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.565651    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.566227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.566205    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnpbl\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-kube-api-access-gnpbl\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.566693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.566668    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.567154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.567137    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.567656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.567637    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f954b829-91a3-4aee-925c-363f1cfa9824-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.568186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.568166    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f954b829-91a3-4aee-925c-363f1cfa9824-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.569032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.569014    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.569539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.569519    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.570007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.568943    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f954b829-91a3-4aee-925c-363f1cfa9824-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.570201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.565578    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.570311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.567563    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.570410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.569893    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.570512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.567088    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.574084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.574016    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f954b829-91a3-4aee-925c-363f1cfa9824-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.576353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.576330    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.581576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.581555    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.586094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.585756    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnpbl\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-kube-api-access-gnpbl\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.642087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.641830    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"]
Nov 03 08:05:15.654084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.648178    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.658084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.656691    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-15a633-openshift-default-dockercfg-vdsjd\""
Nov 03 08:05:15.680086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.677672    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"]
Nov 03 08:05:15.687088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.683323    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:15.806093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.785240    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.806093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.785333    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.806093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.785480    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxmtd\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-kube-api-access-cxmtd\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.806994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.806932    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.807091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.807046    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.807206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.807184    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.807346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.807279    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.807419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.807363    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.807492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.807477    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34142c2c-a6ec-433b-9719-8e7a382fc660-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.840148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.832512    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"]
Nov 03 08:05:15.855094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.853316    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:15.883094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.879764    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"]
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.911747    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.911817    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.911849    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxmtd\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-kube-api-access-cxmtd\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.911916    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.911954    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.911997    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.912033    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.912108    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.912146    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34142c2c-a6ec-433b-9719-8e7a382fc660-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.912950    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34142c2c-a6ec-433b-9719-8e7a382fc660-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.913268    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.918099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.913504    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.958096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.939892    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ffe7a8-openshift-default-dockercfg-cp99l\""
Nov 03 08:05:15.963096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.962829    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:15.995103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:15.987268    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.015176    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.015873    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.015917    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ptd\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-kube-api-access-p6ptd\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.015960    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.016023    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.016092    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.016127    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.016156    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.016204    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.020109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.016233    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138aa05-5757-40de-b2d9-82bce662ec9f-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.142132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.118702    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.142465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.142441    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ptd\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-kube-api-access-p6ptd\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.142616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.142603    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.142763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.142750    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.148501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.142878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.161193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.161163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.161731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.161707    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.161920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.161902    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.162050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.162034    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138aa05-5757-40de-b2d9-82bce662ec9f-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.162328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.127717    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:16.162442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.119724    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.166138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.165386    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.166138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.149236    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.166138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.149473    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.166138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.166095    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138aa05-5757-40de-b2d9-82bce662ec9f-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.171083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.169559    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:16.181083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.180394    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.192084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.183220    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.203084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.202396    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.220084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.210640    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ptd\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-kube-api-access-p6ptd\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.220084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.218880    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxmtd\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-kube-api-access-cxmtd\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:16.287607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.287263    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:16.372088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.371954    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:16.667170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.662231    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"]
Nov 03 08:05:16.671102 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:05:16.670247    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf954b829_91a3_4aee_925c_363f1cfa9824.slice/crio-5e8c91ba54a82cd60277dc62231ad83af25fe090f2e6e2154140aea0e1b90cfd WatchSource:0}: Error finding container 5e8c91ba54a82cd60277dc62231ad83af25fe090f2e6e2154140aea0e1b90cfd: Status 404 returned error can't find the container with id 5e8c91ba54a82cd60277dc62231ad83af25fe090f2e6e2154140aea0e1b90cfd
Nov 03 08:05:16.681210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.681166    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.681488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.681461    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.681665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.681643    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.752254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.752118    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"]
Nov 03 08:05:16.761906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.761218    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.761906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.761304    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.761906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.761348    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.786006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.785980    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"]
Nov 03 08:05:16.789329 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:05:16.789304    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5138aa05_5757_40de_b2d9_82bce662ec9f.slice/crio-b89811cc5b2ba528f6dd32a629d54dcff8a943e31b94a622f986ef6ed1dfedf2 WatchSource:0}: Error finding container b89811cc5b2ba528f6dd32a629d54dcff8a943e31b94a622f986ef6ed1dfedf2: Status 404 returned error can't find the container with id b89811cc5b2ba528f6dd32a629d54dcff8a943e31b94a622f986ef6ed1dfedf2
Nov 03 08:05:16.792042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.792012    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.792266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.792180    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.792358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.792262    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:16.957162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:16.945552    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:17.587152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.586563    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" event={"ID":"5138aa05-5757-40de-b2d9-82bce662ec9f","Type":"ContainerStarted","Data":"840f7db386520802a8db18e041fb4644b652caa4673b05380541632565129671"}
Nov 03 08:05:17.587152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.586639    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" event={"ID":"5138aa05-5757-40de-b2d9-82bce662ec9f","Type":"ContainerStarted","Data":"b89811cc5b2ba528f6dd32a629d54dcff8a943e31b94a622f986ef6ed1dfedf2"}
Nov 03 08:05:17.590618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.590244    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" event={"ID":"34142c2c-a6ec-433b-9719-8e7a382fc660","Type":"ContainerStarted","Data":"c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc"}
Nov 03 08:05:17.590618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.590308    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" event={"ID":"34142c2c-a6ec-433b-9719-8e7a382fc660","Type":"ContainerStarted","Data":"ec8eb45a0f2ef13b9735de703609b87f2c8d2a5892f57f16c906f334e06ff641"}
Nov 03 08:05:17.593187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.593153    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" event={"ID":"f954b829-91a3-4aee-925c-363f1cfa9824","Type":"ContainerStarted","Data":"2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642"}
Nov 03 08:05:17.593295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.593201    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" event={"ID":"f954b829-91a3-4aee-925c-363f1cfa9824","Type":"ContainerStarted","Data":"5e8c91ba54a82cd60277dc62231ad83af25fe090f2e6e2154140aea0e1b90cfd"}
Nov 03 08:05:17.644476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.642939    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podStartSLOduration=2.642920789 podStartE2EDuration="2.642920789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:17.612605699 +0000 UTC m=+667.115441214" watchObservedRunningTime="2026-11-03 08:05:17.642920789 +0000 UTC m=+667.145756304"
Nov 03 08:05:17.677162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.677107    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podStartSLOduration=2.677087806 podStartE2EDuration="2.677087806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:17.644948185 +0000 UTC m=+667.147783700" watchObservedRunningTime="2026-11-03 08:05:17.677087806 +0000 UTC m=+667.179923318"
Nov 03 08:05:17.694687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.694655    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:17.694988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.694965    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:17.695196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.695169    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:17.943602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:17.943568    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:18.294264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.294120    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:18.294264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.294189    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 08:05:18.294264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.294235    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 08:05:18.376315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.376269    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:18.378636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.378592    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 08:05:18.378910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.378877    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 08:05:18.684748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.684702    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:18.685391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.685359    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:18.950154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:18.949774    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:19.287998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:19.287917    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 08:05:19.288836 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:19.288805    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 08:05:19.376630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:19.376593    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 08:05:19.376946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:19.376896    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 08:05:19.684519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:19.684490    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:19.684793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:19.684751    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:19.889088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:19.886759    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:20.290128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:20.290085    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 08:05:20.290332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:20.290144    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 08:05:20.377088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:20.376457    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 08:05:20.377088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:20.376522    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 08:05:20.672778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:20.672692    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:20.685095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:20.684777    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:20.685095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:20.684833    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:21.288574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.288536    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 08:05:21.288750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.288603    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 08:05:21.375095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.373488    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 08:05:21.375095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.373549    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 08:05:21.430298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.430230    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podStartSLOduration=6.430209222 podStartE2EDuration="6.430209222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:17.677908008 +0000 UTC m=+667.180743523" watchObservedRunningTime="2026-11-03 08:05:21.430209222 +0000 UTC m=+670.933044739"
Nov 03 08:05:21.433002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.432966    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"]
Nov 03 08:05:21.448369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.448339    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.457091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.453391    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"]
Nov 03 08:05:21.475842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.475727    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.477622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.477596    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.477861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.477844    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.477974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.477960    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.478130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.478114    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.478278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.478264    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rh6\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-kube-api-access-85rh6\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.478463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.478447    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.497223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.497190    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.497479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.497463    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.587192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.587107    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:21.602718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.602677    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.611372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.611342    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.612108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.612087    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.680269    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rh6\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-kube-api-access-85rh6\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.680431    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.680489    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.680535    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.680583    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.680608    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.635606    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.611999    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.681514    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.685105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.681823    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.702087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.696790    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:21.771396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.771344    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:21.785087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.784398    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rh6\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-kube-api-access-85rh6\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.785087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.697621    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.785087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.735429    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.786688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.786658    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.787381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.787361    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bxgs4\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:21.846141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:21.839409    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:22.299087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.289513    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 08:05:22.299087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.289573    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 08:05:22.396088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.394085    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 08:05:22.396088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.394142    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 08:05:22.408093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.406409    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"]
Nov 03 08:05:22.532139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.522275    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:22.532139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.522358    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:22.532139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.522397    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:22.643395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.643318    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:22.684041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.684010    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:22.691300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.691176    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:22.950089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:22.946687    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" event={"ID":"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5","Type":"ContainerStarted","Data":"93255979eef29fe9e6be3190e09e8ed390da46a8f9e4f9353c88e0eb9f15e309"}
Nov 03 08:05:23.292413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.292312    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 08:05:23.292641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.292615    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 08:05:23.376730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.376684    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 08:05:23.377185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.377104    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 08:05:23.500180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.500144    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:23.684608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.684579    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:23.684875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.684839    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:23.770901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.770721    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:23.907102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.903172    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:23.988136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.935476    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:23.988136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:23.965049    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:24.027128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.014943    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:24.112113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.107488    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" event={"ID":"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5","Type":"ContainerStarted","Data":"e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894"}
Nov 03 08:05:24.178396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.178334    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podStartSLOduration=3.178313009 podStartE2EDuration="3.178313009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:24.175701277 +0000 UTC m=+673.678536787" watchObservedRunningTime="2026-11-03 08:05:24.178313009 +0000 UTC m=+673.681148524"
Nov 03 08:05:24.290451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.287976    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 08:05:24.290451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.288046    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 08:05:24.291560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.291536    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:24.373314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.373278    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 08:05:24.373459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.373346    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 08:05:24.461565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.461517    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:24.684593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.684551    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 08:05:24.684785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.684637    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 08:05:24.854486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.854415    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:24.856753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.856717    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:24.856910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:24.856778    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:25.080559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.080257    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:05:25.353337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.350790    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:25.528124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.527570    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:25.542083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.528986    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:25.552629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.552560    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:05:25.637484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.637393    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:25.642653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.642617    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:25.647580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.646135    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:05:25.780512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.780484    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:25.781265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.781245    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:25.782182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.782142    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:05:25.842479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.842443    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:25.842629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:25.842503    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:26.071092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:26.071049    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:26.844798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:26.843480    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:26.844798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:26.843561    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:26.983110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:26.983055    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:27.841720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:27.841683    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:27.842349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:27.842313    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:28.504088 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:05:28.495384    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:05:28.637084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:28.636448    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:28.842146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:28.841604    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:28.842146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:28.841703    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:29.552529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:29.552499    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:29.840896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:29.840802    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:29.840896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:29.840868    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:30.424345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.424276    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:30.424863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.424538    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"]
Nov 03 08:05:30.429801 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.429769    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.448215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.448185    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-bd085d-openshift-default-dockercfg-v65xr\""
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.458902    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d752af-c375-437c-bde9-7886542bf533-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.458967    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.459030    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.459093    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgp5\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-kube-api-access-4sgp5\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.459130    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.459178    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.459222    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.459254    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.459527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.459314    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d752af-c375-437c-bde9-7886542bf533-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.469104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.467046    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"]
Nov 03 08:05:30.560417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.560386    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"]
Nov 03 08:05:30.562931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.562893    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d752af-c375-437c-bde9-7886542bf533-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.563101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.562988    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d752af-c375-437c-bde9-7886542bf533-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.563101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.563029    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.563727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.563695    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.563891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.563758    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d752af-c375-437c-bde9-7886542bf533-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.565082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.565042    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.565258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.565237    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgp5\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-kube-api-access-4sgp5\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.565348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.565282    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.565348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.565329    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.565453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.565381    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.565453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.565412    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.565777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.565752    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.566409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.566386    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.567122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.566731    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d752af-c375-437c-bde9-7886542bf533-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.567363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.567223    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.570385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.570365    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.570495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.570440    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.583416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.583390    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"]
Nov 03 08:05:30.593864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.593837    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.595912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.595888    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgp5\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-kube-api-access-4sgp5\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.667041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.666986    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667200    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667325    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kjb\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-kube-api-access-62kjb\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667464    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667547    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667624    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667677    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667701    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.667759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.667721    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.768778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.768746    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.768778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.768779    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kjb\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-kube-api-access-62kjb\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.768807    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.768847    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769000    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769037    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769083    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769119    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769360    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769698    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769703    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.769898 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.769821    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.770314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.770288    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.771730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.771705    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.772289 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.772267    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.781930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.781415    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:30.784195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.784170    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kjb\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-kube-api-access-62kjb\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.786795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.786763    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:30.841168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.841133    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:30.841320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.841183    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:30.935865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:30.935654    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:31.084965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.084933    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"]
Nov 03 08:05:31.087141 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:05:31.087110    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d752af_c375_437c_bde9_7886542bf533.slice/crio-ef0034f8820bdc7b6221086ab22a624a8449d4296bd9c3b805715701113acc60 WatchSource:0}: Error finding container ef0034f8820bdc7b6221086ab22a624a8449d4296bd9c3b805715701113acc60: Status 404 returned error can't find the container with id ef0034f8820bdc7b6221086ab22a624a8449d4296bd9c3b805715701113acc60
Nov 03 08:05:31.090342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.090303    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:31.090444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.090386    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:31.090444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.090431    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:31.199733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.199625    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" event={"ID":"b6d752af-c375-437c-bde9-7886542bf533","Type":"ContainerStarted","Data":"8864974f72ce13924b4effac8982a8286738963642423b888270e76b4b5c0a4e"}
Nov 03 08:05:31.199733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.199693    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" event={"ID":"b6d752af-c375-437c-bde9-7886542bf533","Type":"ContainerStarted","Data":"ef0034f8820bdc7b6221086ab22a624a8449d4296bd9c3b805715701113acc60"}
Nov 03 08:05:31.247933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.247865    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podStartSLOduration=1.247843579 podStartE2EDuration="1.247843579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:31.225963372 +0000 UTC m=+680.728798896" watchObservedRunningTime="2026-11-03 08:05:31.247843579 +0000 UTC m=+680.750679095"
Nov 03 08:05:31.251289 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.250999    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"]
Nov 03 08:05:31.254700 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:05:31.254674    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2366709e_fe4d_4cbd_8aa7_ec0f618b4a48.slice/crio-7f36810fbf30812dd5d14b3afa64523571b9ead62b07b23f420d6519da3ffb22 WatchSource:0}: Error finding container 7f36810fbf30812dd5d14b3afa64523571b9ead62b07b23f420d6519da3ffb22: Status 404 returned error can't find the container with id 7f36810fbf30812dd5d14b3afa64523571b9ead62b07b23f420d6519da3ffb22
Nov 03 08:05:31.257708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.257669    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:31.257800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.257735    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:31.257800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.257764    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:05:31.344400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.344301    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:31.783520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.783481    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:31.841199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.841167    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 08:05:31.841352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:31.841220    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 08:05:32.229166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.229130    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" event={"ID":"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48","Type":"ContainerStarted","Data":"38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd"}
Nov 03 08:05:32.230342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.230314    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" event={"ID":"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48","Type":"ContainerStarted","Data":"7f36810fbf30812dd5d14b3afa64523571b9ead62b07b23f420d6519da3ffb22"}
Nov 03 08:05:32.285088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.283494    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podStartSLOduration=2.283471379 podStartE2EDuration="2.283471379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:32.272512423 +0000 UTC m=+681.775347944" watchObservedRunningTime="2026-11-03 08:05:32.283471379 +0000 UTC m=+681.786306898"
Nov 03 08:05:32.522718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.522598    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:32.786859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.786782    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 08:05:32.787381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.786856    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 08:05:32.881573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.881526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:32.984039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.984007    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:32.985142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.984988    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 08:05:32.985142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:32.985043    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 08:05:33.791084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:33.783674    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 08:05:33.791084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:33.783751    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 08:05:33.884302 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:33.884265    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bxgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:05:33.884576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:33.884533    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:33.967195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:33.961573    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:33.967195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:33.961725    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 08:05:33.967195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:33.961773    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 08:05:34.776997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:34.776968    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:34.777527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:34.777323    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:34.778286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:34.778264    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:05:34.784255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:34.784229    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 08:05:34.784363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:34.784297    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 08:05:34.942088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:34.939431    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 08:05:34.942088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:34.939490    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 08:05:35.356003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:35.351856    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:35.783776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:35.783743    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 08:05:35.783930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:35.783792    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 08:05:35.942084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:35.942030    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 08:05:35.942357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:35.942327    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 08:05:36.340087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:36.339805    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:36.783644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:36.783609    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 08:05:36.783814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:36.783662    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 08:05:36.940088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:36.939410    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 08:05:36.940088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:36.939481    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 08:05:37.690340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:37.690309    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:37.796126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:37.789269    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 08:05:37.796126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:37.789342    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 08:05:37.938435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:37.938392    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 08:05:37.938639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:37.938468    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 08:05:38.341684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.341646    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:38.729492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.729447    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:38.922237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.922203    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:38.923931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.923902    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:38.925108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.925087    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:05:38.955337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.955305    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:38.955620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.955597    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:38.956356 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:38.956338    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:05:39.656967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:39.656932    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:40.509130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:40.508516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:41.031117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:41.030607    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:41.257888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:41.257853    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:42.045785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:42.045709    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:42.857429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:42.857397    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:43.437377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.435355    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"]
Nov 03 08:05:43.495394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.485042    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"]
Nov 03 08:05:43.495394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.485219    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.495394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.489760    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"distinct-w3-3e88b7-kserve-self-signed-certs\""
Nov 03 08:05:43.551677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.551559    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:43.620960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.620543    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-home\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.620960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.620635    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-tmp-dir\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.620960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.620697    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-dshm\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.620960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.620731    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba467c2-f014-4052-8ebc-e769cb311877-tls-certs\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.620960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.620756    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q6h\" (UniqueName: \"kubernetes.io/projected/6ba467c2-f014-4052-8ebc-e769cb311877-kube-api-access-h6q6h\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.620960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.620788    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-model-cache\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.722730    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-home\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.722845    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-tmp-dir\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.722904    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-dshm\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.722941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba467c2-f014-4052-8ebc-e769cb311877-tls-certs\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.722970    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q6h\" (UniqueName: \"kubernetes.io/projected/6ba467c2-f014-4052-8ebc-e769cb311877-kube-api-access-h6q6h\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.723013    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-model-cache\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.723623    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-model-cache\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.723874    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-home\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.724522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.724146    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-tmp-dir\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.745911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.745880    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-dshm\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.764098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.758794    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q6h\" (UniqueName: \"kubernetes.io/projected/6ba467c2-f014-4052-8ebc-e769cb311877-kube-api-access-h6q6h\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.789100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.783797    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba467c2-f014-4052-8ebc-e769cb311877-tls-certs\") pod \"distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:43.874315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:43.874276    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:44.191641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:44.173707    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:44.772089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:44.770712    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"]
Nov 03 08:05:45.546677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:45.546635    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:45.617753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:45.617714    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" event={"ID":"6ba467c2-f014-4052-8ebc-e769cb311877","Type":"ContainerStarted","Data":"a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9"}
Nov 03 08:05:45.618036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:45.618004    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" event={"ID":"6ba467c2-f014-4052-8ebc-e769cb311877","Type":"ContainerStarted","Data":"e16effe085ba5ef284d714766965bc5dd7af5e6e4cb0ccac82684477bc8b9bfa"}
Nov 03 08:05:45.646117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:45.646037    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podStartSLOduration=2.646018319 podStartE2EDuration="2.646018319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:45.644317519 +0000 UTC m=+695.147153046" watchObservedRunningTime="2026-11-03 08:05:45.646018319 +0000 UTC m=+695.148853839"
Nov 03 08:05:45.877915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:45.877835    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:46.105946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:46.105906    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:46.549223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:46.549184    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:47.464109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:47.464078    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:48.376856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:48.376823    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:49.303087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.303030    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"]
Nov 03 08:05:49.308616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.308591    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.316901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.316723    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"distinct-2-w5-ffe7a8-kserve-self-signed-certs\""
Nov 03 08:05:49.320262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.320234    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"]
Nov 03 08:05:49.372407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.372375    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:49.382690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.382662    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-tmp-dir\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.382832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.382703    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc94a01-1373-403b-b51c-cc7789cab2d2-tls-certs\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.382967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.382945    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-dshm\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.383115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.383093    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5rv\" (UniqueName: \"kubernetes.io/projected/dcc94a01-1373-403b-b51c-cc7789cab2d2-kube-api-access-fv5rv\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.383173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.383162    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-home\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.383249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.383229    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-model-cache\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.480970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.480937    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:49.484324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.484297    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc94a01-1373-403b-b51c-cc7789cab2d2-tls-certs\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.484487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.484456    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-dshm\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.484627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.484546    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5rv\" (UniqueName: \"kubernetes.io/projected/dcc94a01-1373-403b-b51c-cc7789cab2d2-kube-api-access-fv5rv\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.484627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.484594    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-home\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.484627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.484621    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-model-cache\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.484803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.484667    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-tmp-dir\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.485271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.485025    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-tmp-dir\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.485271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.485085    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-home\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.485271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.485230    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-model-cache\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.486888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.486866    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-dshm\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.487272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.487251    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc94a01-1373-403b-b51c-cc7789cab2d2-tls-certs\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.492187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.492163    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5rv\" (UniqueName: \"kubernetes.io/projected/dcc94a01-1373-403b-b51c-cc7789cab2d2-kube-api-access-fv5rv\") pod \"distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") " pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.628475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.627272    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:49.932396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:49.932227    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"]
Nov 03 08:05:49.935486 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:05:49.935461    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc94a01_1373_403b_b51c_cc7789cab2d2.slice/crio-84a577b700f40ad2108adf81d213b12c7e74303c147885c2a46f3bfc1faffe93 WatchSource:0}: Error finding container 84a577b700f40ad2108adf81d213b12c7e74303c147885c2a46f3bfc1faffe93: Status 404 returned error can't find the container with id 84a577b700f40ad2108adf81d213b12c7e74303c147885c2a46f3bfc1faffe93
Nov 03 08:05:50.418658 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:50.418543    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:50.655713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:50.655679    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" event={"ID":"dcc94a01-1373-403b-b51c-cc7789cab2d2","Type":"ContainerStarted","Data":"b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4"}
Nov 03 08:05:50.655893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:50.655720    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" event={"ID":"dcc94a01-1373-403b-b51c-cc7789cab2d2","Type":"ContainerStarted","Data":"84a577b700f40ad2108adf81d213b12c7e74303c147885c2a46f3bfc1faffe93"}
Nov 03 08:05:50.671841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:50.671747    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podStartSLOduration=1.6717299140000002 podStartE2EDuration="1.671729914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:50.669743452 +0000 UTC m=+700.172578967" watchObservedRunningTime="2026-11-03 08:05:50.671729914 +0000 UTC m=+700.174565427"
Nov 03 08:05:51.341788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:51.341726    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:51.628633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:51.628541    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:52.412032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:52.412001    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:53.357961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:53.357923    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:53.874710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:53.874672    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:53.887736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:53.887709    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:54.265519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:54.265486    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:54.689308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:54.689277    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:05:55.039766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:55.039675    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:55.759296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:55.759265    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:56.601254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:56.601223    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:57.599045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:57.599006    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:58.577372 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:05:58.577321    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:05:59.207973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:59.207942    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:59.587551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:59.587460    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:05:59.628514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:59.628480    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:59.642534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:59.642499    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:05:59.800494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:05:59.800437    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:06:00.393859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.393829    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"]
Nov 03 08:06:00.397962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.397933    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.411316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.411292    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"]
Nov 03 08:06:00.431478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431445    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.431647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431601    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.431647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431632    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.431795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431714    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.431795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431755    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlpb\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-kube-api-access-qqlpb\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.431907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431801    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.431907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431833    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e103cd-753b-4ca4-be50-7a17dfe55440-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.431907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431892    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.432055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.431980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.460450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.460422    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:00.538084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.537971    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538022    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538117    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538148    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlpb\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-kube-api-access-qqlpb\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538220    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538249    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e103cd-753b-4ca4-be50-7a17dfe55440-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538303    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538385    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.538687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.538438    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.545116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.539055    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.545116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.539331    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.545116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.539916    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.545890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.545683    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.546663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.546639    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e103cd-753b-4ca4-be50-7a17dfe55440-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.560672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.554488    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.560672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.554614    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.560672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.556660    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.560672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.558372    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlpb\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-kube-api-access-qqlpb\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.714668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.714626    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:00.717341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.717317    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"]
Nov 03 08:06:00.720944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.720930    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.734208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.734182    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"]
Nov 03 08:06:00.842566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.842528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.842566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.842567    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.842805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.842719    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.842805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.842777    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.842916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.842825    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.843031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.843012    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.843081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.843074    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vjqp\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-kube-api-access-4vjqp\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.843119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.843093    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57e36a8-937e-44f0-b243-a95a341901a0-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.843406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.843389    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57e36a8-937e-44f0-b243-a95a341901a0-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944160    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944214    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vjqp\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-kube-api-access-4vjqp\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944235    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57e36a8-937e-44f0-b243-a95a341901a0-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944281    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57e36a8-937e-44f0-b243-a95a341901a0-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944344    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944367    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944398    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944440    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944483    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944900    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.944989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.944949    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.945126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.945099    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.945477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.945384    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57e36a8-937e-44f0-b243-a95a341901a0-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.945553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.945529    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.947364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.947344    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57e36a8-937e-44f0-b243-a95a341901a0-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.947364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.947358    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.951197 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.951172    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:00.951483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:00.951465    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vjqp\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-kube-api-access-4vjqp\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:01.062763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.062689    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:01.083766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.083739    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"]
Nov 03 08:06:01.085976 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:01.085933    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e103cd_753b_4ca4_be50_7a17dfe55440.slice/crio-daf39b5ba9ff30f7e04610aff5c8358d829cc4b67e45c91bb90dc23ee755f839 WatchSource:0}: Error finding container daf39b5ba9ff30f7e04610aff5c8358d829cc4b67e45c91bb90dc23ee755f839: Status 404 returned error can't find the container with id daf39b5ba9ff30f7e04610aff5c8358d829cc4b67e45c91bb90dc23ee755f839
Nov 03 08:06:01.088909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.088872    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:01.089019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.088959    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:01.089019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.089003    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:01.353508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.353480    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"]
Nov 03 08:06:01.355819 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:01.355767    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57e36a8_937e_44f0_b243_a95a341901a0.slice/crio-9ce5723efd4db42fae045b2915d2f6f6d58a6d81c093eb2ff6ff14212dd342aa WatchSource:0}: Error finding container 9ce5723efd4db42fae045b2915d2f6f6d58a6d81c093eb2ff6ff14212dd342aa: Status 404 returned error can't find the container with id 9ce5723efd4db42fae045b2915d2f6f6d58a6d81c093eb2ff6ff14212dd342aa
Nov 03 08:06:01.359011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.358974    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:01.359127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.359082    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:01.359173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.359128    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:01.505333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.505294    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:01.775041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.774994    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" event={"ID":"80e103cd-753b-4ca4-be50-7a17dfe55440","Type":"ContainerStarted","Data":"13c1f47c243249528f3d051efae7a9387397bc51c145d14b2578bcfa7105176d"}
Nov 03 08:06:01.775257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.775084    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" event={"ID":"80e103cd-753b-4ca4-be50-7a17dfe55440","Type":"ContainerStarted","Data":"daf39b5ba9ff30f7e04610aff5c8358d829cc4b67e45c91bb90dc23ee755f839"}
Nov 03 08:06:01.776936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.776910    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" event={"ID":"e57e36a8-937e-44f0-b243-a95a341901a0","Type":"ContainerStarted","Data":"69a9a82ee685a1c37129e135e368afe0c89ae2256285a534c6349262c5e3dc16"}
Nov 03 08:06:01.777040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.776952    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" event={"ID":"e57e36a8-937e-44f0-b243-a95a341901a0","Type":"ContainerStarted","Data":"9ce5723efd4db42fae045b2915d2f6f6d58a6d81c093eb2ff6ff14212dd342aa"}
Nov 03 08:06:01.797734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.797676    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podStartSLOduration=1.7976561100000001 podStartE2EDuration="1.79765611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:01.793557635 +0000 UTC m=+711.296393151" watchObservedRunningTime="2026-11-03 08:06:01.79765611 +0000 UTC m=+711.300491625"
Nov 03 08:06:01.812708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:01.812660    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podStartSLOduration=1.812648785 podStartE2EDuration="1.812648785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:01.811664548 +0000 UTC m=+711.314500062" watchObservedRunningTime="2026-11-03 08:06:01.812648785 +0000 UTC m=+711.315484299"
Nov 03 08:06:02.063011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:02.062934    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:02.064892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:02.064869    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 08:06:02.064960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:02.064925    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 08:06:02.363882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:02.363808    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:02.721338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:02.721252    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:02.723572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:02.723544    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:06:02.723796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:02.723772    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:06:03.063030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:03.062958    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 08:06:03.063030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:03.063007    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 08:06:03.450972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:03.450929    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:03.726169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:03.723444    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:06:03.726169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:03.723517    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:06:04.063470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:04.063397    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 08:06:04.063684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:04.063661    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 08:06:04.381203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:04.381103    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:04.722086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:04.721085    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:06:04.722086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:04.721139    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:06:05.063670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:05.063594    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 08:06:05.063670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:05.063639    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 08:06:05.480882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:05.480833    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:05.730086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:05.729435    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:06:05.730086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:05.729492    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:06:06.064089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:06.063035    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 08:06:06.064089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:06.063124    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 08:06:06.456005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:06.455947    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:06.717166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:06.716508    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:06:06.717166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:06.716567    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:06:07.063396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.063312    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 08:06:07.063396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.063364    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 08:06:07.424135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.421112    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:07.715335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.715300    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:06:07.715538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.715367    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:06:07.769263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.769224    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"]
Nov 03 08:06:07.794475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.794447    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:07.798913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.795828    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"]
Nov 03 08:06:07.800102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.800079    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"distinct-2-w6-bd085d-kserve-self-signed-certs\""
Nov 03 08:06:07.852376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.852322    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:07.926446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.926410    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-home\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:07.944128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.927987    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tls-certs\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:07.944834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.944803    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-model-cache\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:07.945226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.945160    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-dshm\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:07.945458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.945441    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tmp-dir\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:07.945618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:07.945600    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctfj\" (UniqueName: \"kubernetes.io/projected/bba4e9c9-971e-4be9-80db-813f7e1a70ee-kube-api-access-fctfj\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.049142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.049005    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tls-certs\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.049142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.049102    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-model-cache\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.049379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.049193    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-dshm\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.049379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.049250    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tmp-dir\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.049379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.049284    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctfj\" (UniqueName: \"kubernetes.io/projected/bba4e9c9-971e-4be9-80db-813f7e1a70ee-kube-api-access-fctfj\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.049528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.049390    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-home\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.049876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.049844    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-home\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.050141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.050120    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tmp-dir\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.050501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.050481    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-model-cache\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.064547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.064513    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 08:06:08.064821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.064791    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 08:06:08.071874 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.071844    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctfj\" (UniqueName: \"kubernetes.io/projected/bba4e9c9-971e-4be9-80db-813f7e1a70ee-kube-api-access-fctfj\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.072698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.072671    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tls-certs\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.085964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.084995    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-dshm\") pod \"distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.119303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.119267    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:08.431448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.431417    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"]
Nov 03 08:06:08.433454 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:08.433412    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbba4e9c9_971e_4be9_80db_813f7e1a70ee.slice/crio-0278d10531d1db951a2b1d11d6cf30e10b4d4518c549c4425d3e8679b1d1efc0 WatchSource:0}: Error finding container 0278d10531d1db951a2b1d11d6cf30e10b4d4518c549c4425d3e8679b1d1efc0: Status 404 returned error can't find the container with id 0278d10531d1db951a2b1d11d6cf30e10b4d4518c549c4425d3e8679b1d1efc0
Nov 03 08:06:08.449908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.449874    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:08.863482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.863443    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" event={"ID":"bba4e9c9-971e-4be9-80db-813f7e1a70ee","Type":"ContainerStarted","Data":"e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5"}
Nov 03 08:06:08.863777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.863744    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" event={"ID":"bba4e9c9-971e-4be9-80db-813f7e1a70ee","Type":"ContainerStarted","Data":"0278d10531d1db951a2b1d11d6cf30e10b4d4518c549c4425d3e8679b1d1efc0"}
Nov 03 08:06:08.878631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:08.878550    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podStartSLOduration=1.878532178 podStartE2EDuration="1.878532178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:08.877597973 +0000 UTC m=+718.380433490" watchObservedRunningTime="2026-11-03 08:06:08.878532178 +0000 UTC m=+718.381367697"
Nov 03 08:06:09.105120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.104653    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:09.105798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.105767    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:09.108642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.107604    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:06:09.120346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.120281    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:09.244539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.244506    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:09.246255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.246229    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:09.248008 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.247983    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:06:09.330840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:09.330811    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:10.052968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:10.052935    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:10.737611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:10.737576    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:11.465946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:11.465903    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:11.648845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:11.648811    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:11.770137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:11.760419    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:12.327214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:12.327181    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:13.109682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:13.109646    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:13.861218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:13.861178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:14.571903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:14.571861    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:15.358526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.358457    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:15.495085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.494249    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"]
Nov 03 08:06:15.510085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.500869    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.532085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.525789    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"]
Nov 03 08:06:15.548086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.546693    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"]
Nov 03 08:06:15.557250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.557222    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.564139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.564028    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"]
Nov 03 08:06:15.651899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.651828    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"]
Nov 03 08:06:15.656487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.656436    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.689808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.689784    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.689960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.689839    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.689960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.689930    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.690098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.689969    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.690098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690033    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690089    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690121    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pn98\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-kube-api-access-4pn98\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690169    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690185    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.690245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690210    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690242    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690267    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.690500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690305    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.690500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690352    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c4fcc3-a069-4c7c-9d32-bf475a418747-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.690500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690373    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690396    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.690500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690477    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnqw\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-kube-api-access-lbnqw\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.690746 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.690548    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.701002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.700976    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"]
Nov 03 08:06:15.791453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791406    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.791453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791454    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.791697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791492    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.791697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791550    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.791697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791590    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.791697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791619    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c4fcc3-a069-4c7c-9d32-bf475a418747-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.791697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791687    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791717    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791747    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnqw\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-kube-api-access-lbnqw\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791775    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791801    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791849    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791877    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791897    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.791957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791916    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7slj\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-kube-api-access-h7slj\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791982    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792026    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792054    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792140    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792200    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.791978    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792297    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792330    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.792384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792339    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.792748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792480    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.792748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792631    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.792748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792672    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.792862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792763    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.792862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792785    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pn98\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-kube-api-access-4pn98\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.792862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792819    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.792862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792846    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.793000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792973    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.793000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.792984    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c4fcc3-a069-4c7c-9d32-bf475a418747-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.793089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.793004    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.793279 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.793258    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.793350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.793258    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.793650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.793634    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.794785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.794759    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.794897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.794764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.795049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.795029    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.795890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.795872    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.798711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.798687    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.798972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.798953    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnqw\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-kube-api-access-lbnqw\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.799540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.799520    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.800009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.799991    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pn98\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-kube-api-access-4pn98\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.894206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894179    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894215    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894235    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894281    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894306    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894442    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894489    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894531    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894553    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894565    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7slj\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-kube-api-access-h7slj\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894639    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.894862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.894786    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.895098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.895008    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.895098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.895202    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.896970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.896948    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.897126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.897054    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.901107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.901045    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.901690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.901667    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7slj\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-kube-api-access-h7slj\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:15.904433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.904381    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:15.913031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.913012    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:15.972928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:15.972891    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:16.210284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.210166    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"]
Nov 03 08:06:16.212430 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:16.212406    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd2af55_fc63_4e95_9a95_6c2a2e8fdcfe.slice/crio-69ee7a9d1722034b1d03f5595ef56f470757f3ea143a7c9af780ab0602fafdb6 WatchSource:0}: Error finding container 69ee7a9d1722034b1d03f5595ef56f470757f3ea143a7c9af780ab0602fafdb6: Status 404 returned error can't find the container with id 69ee7a9d1722034b1d03f5595ef56f470757f3ea143a7c9af780ab0602fafdb6
Nov 03 08:06:16.214600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.214530    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"]
Nov 03 08:06:16.215093 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:16.215039    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c4fcc3_a069_4c7c_9d32_bf475a418747.slice/crio-45825be3f66b8f70ed577c5654466e76008720b5bcc5e908369c664d31f92191 WatchSource:0}: Error finding container 45825be3f66b8f70ed577c5654466e76008720b5bcc5e908369c664d31f92191: Status 404 returned error can't find the container with id 45825be3f66b8f70ed577c5654466e76008720b5bcc5e908369c664d31f92191
Nov 03 08:06:16.215640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.215573    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.215697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.215681    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.215738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.215729    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.217611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.217586    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.217711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.217642    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.217711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.217678    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.284951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.284665    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"]
Nov 03 08:06:16.291907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.291874    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.292014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.291952    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.292014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.291996    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:16.369600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.369573    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:16.497219 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.497017    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:16.946408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.946375    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" event={"ID":"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd","Type":"ContainerStarted","Data":"b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d"}
Nov 03 08:06:16.946904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.946882    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" event={"ID":"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd","Type":"ContainerStarted","Data":"ff33857a796e523d21ce17b89b071845520b9d920b48835b42d385d19665938b"}
Nov 03 08:06:16.948238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.947944    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" event={"ID":"34c4fcc3-a069-4c7c-9d32-bf475a418747","Type":"ContainerStarted","Data":"041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3"}
Nov 03 08:06:16.949830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.949801    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" event={"ID":"34c4fcc3-a069-4c7c-9d32-bf475a418747","Type":"ContainerStarted","Data":"45825be3f66b8f70ed577c5654466e76008720b5bcc5e908369c664d31f92191"}
Nov 03 08:06:16.950268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.950208    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" event={"ID":"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe","Type":"ContainerStarted","Data":"68516de526dea87710aff4b3bf9ab4c8c384385883384204eaab05494785dc84"}
Nov 03 08:06:16.950376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.950300    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" event={"ID":"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe","Type":"ContainerStarted","Data":"69ee7a9d1722034b1d03f5595ef56f470757f3ea143a7c9af780ab0602fafdb6"}
Nov 03 08:06:16.965807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.965764    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podStartSLOduration=1.965752552 podStartE2EDuration="1.965752552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:16.962359434 +0000 UTC m=+726.465194948" watchObservedRunningTime="2026-11-03 08:06:16.965752552 +0000 UTC m=+726.468588065"
Nov 03 08:06:16.973580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.973544    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:16.979286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.979234    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podStartSLOduration=1.979217467 podStartE2EDuration="1.979217467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:16.977858657 +0000 UTC m=+726.480694171" watchObservedRunningTime="2026-11-03 08:06:16.979217467 +0000 UTC m=+726.482053008"
Nov 03 08:06:16.997653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:16.997612    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podStartSLOduration=1.997600261 podStartE2EDuration="1.997600261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:16.994048403 +0000 UTC m=+726.496883916" watchObservedRunningTime="2026-11-03 08:06:16.997600261 +0000 UTC m=+726.500435775"
Nov 03 08:06:17.480215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.480181    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:17.912228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.912136    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:17.912990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.912963    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:06:17.913208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.913183    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:06:17.914831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.914811    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:17.915046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.915031    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:06:17.915195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.915175    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 08:06:17.979031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.978997    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:06:17.980286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:17.980244    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 08:06:18.124085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.120461    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:18.184478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.184452    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:18.657548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.657464    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:18.914139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.908466    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:06:18.914139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.908529    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:06:18.914139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.913446    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:06:18.914139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.913495    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 08:06:18.974086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.973468    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:06:18.974086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:18.973525    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 08:06:19.086173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.086138    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:19.750146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.749233    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:19.915480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.915443    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:06:19.917512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.917479    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:06:19.921510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.915771    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:06:19.925117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.921685    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 08:06:19.990085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.989694    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:06:19.990085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:19.989760    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 08:06:20.834270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:20.834218    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:20.970553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:20.955871    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:06:20.970553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:20.955926    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:06:20.987088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:20.973641    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:06:20.988927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:20.988887    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 08:06:20.989324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:20.980526    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:06:20.989893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:20.989872    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 08:06:21.773884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:21.773850    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:21.906602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:21.906567    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:06:21.907274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:21.907240    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:06:21.916299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:21.916272    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:06:21.916496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:21.916472    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 08:06:21.974602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:21.974570    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:06:21.974925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:21.974879    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 08:06:22.917513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:22.917474    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:06:22.918186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:22.918136    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:06:22.918856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:22.918828    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:06:22.919013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:22.918993    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 08:06:22.920838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:22.920665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:22.980087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:22.975797    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:06:22.980087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:22.975867    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 08:06:23.641777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:23.641740    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:23.905346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:23.905257    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:06:23.905346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:23.905320    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:06:23.913824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:23.913794    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:06:23.914029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:23.914005    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 08:06:23.974134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:23.974100    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:06:23.974730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:23.974697    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 08:06:24.596087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:24.595560    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:25.150518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.150478    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:25.152458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.151403    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:25.152458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.152388    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:06:25.242354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.242323    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:25.244051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.244014    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:25.245564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.245546    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:06:25.254127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.254095    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:25.273147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.255687    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:25.282625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.282590    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:06:25.541324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:25.541286    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:27.688101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:27.684154    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:27.865218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:27.865169    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:27.880087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:27.875373    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:27.920088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:27.919304    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:27.963145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:27.962157    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.025094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.024168    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.029084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.025368    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.046085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.034257    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:06:28.087090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.086238    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.590347 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:06:28.590291    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:06:28.693323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.692260    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:06:28.786804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.783873    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:06:28.786804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.786638    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:06:28.971165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.951332    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.971165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.966309    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:06:28.971165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:28.968002    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:29.011709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.011669    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:06:29.080093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.062238    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:29.138099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.081364    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:29.138099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.118148    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:29.138099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.118214    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:06:29.138099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.122571    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:29.551979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.551947    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:06:29.770588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.770543    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:29.818821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.786315    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:06:29.818821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.787450    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:06:29.993259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:29.993222    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:30.430461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.430429    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"]
Nov 03 08:06:30.434630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.434609    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.446472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.446442    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"]
Nov 03 08:06:30.507491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507458    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.507685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507522    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.507685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507611    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e73f76-be64-4daf-89ec-09cbea76457c-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.507685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507679    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e73f76-be64-4daf-89ec-09cbea76457c-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.507861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507804    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.507861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507836    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.507966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507908    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.508022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507963    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n57b5\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-kube-api-access-n57b5\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.508022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.507997    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.614025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.613991    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e73f76-be64-4daf-89ec-09cbea76457c-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.614331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.614310    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.614471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.614454    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.614859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.614837    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.615014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.614994    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n57b5\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-kube-api-access-n57b5\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.615163 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.615146    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.615409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.615388    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.615582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.615557    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.615755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.615724    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e73f76-be64-4daf-89ec-09cbea76457c-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.616019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.616000    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.616713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.616680    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.616972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.616950    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.617049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.617015    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.617491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.617463    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e73f76-be64-4daf-89ec-09cbea76457c-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.619303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.619275    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.619687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.619668    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e73f76-be64-4daf-89ec-09cbea76457c-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.625179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.625156    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.625376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.625360    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n57b5\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-kube-api-access-n57b5\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:30.753994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:30.753963    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:31.042683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.042599    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"]
Nov 03 08:06:31.046145 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:31.046109    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e73f76_be64_4daf_89ec_09cbea76457c.slice/crio-ff6a9118a008261ba5b4dc5911f102b61a83fc64509c5899ce8bb51fbeab4963 WatchSource:0}: Error finding container ff6a9118a008261ba5b4dc5911f102b61a83fc64509c5899ce8bb51fbeab4963: Status 404 returned error can't find the container with id ff6a9118a008261ba5b4dc5911f102b61a83fc64509c5899ce8bb51fbeab4963
Nov 03 08:06:31.048708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.048673    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:31.048800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.048751    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:31.048800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.048793    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:31.364526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.364433    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" event={"ID":"13e73f76-be64-4daf-89ec-09cbea76457c","Type":"ContainerStarted","Data":"bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17"}
Nov 03 08:06:31.364526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.364498    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" event={"ID":"13e73f76-be64-4daf-89ec-09cbea76457c","Type":"ContainerStarted","Data":"ff6a9118a008261ba5b4dc5911f102b61a83fc64509c5899ce8bb51fbeab4963"}
Nov 03 08:06:31.384910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.384860    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podStartSLOduration=1.384846425 podStartE2EDuration="1.384846425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:31.382794547 +0000 UTC m=+740.885630071" watchObservedRunningTime="2026-11-03 08:06:31.384846425 +0000 UTC m=+740.887681939"
Nov 03 08:06:31.755158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:31.755124    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:32.336277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:32.336224    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:32.756112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:32.756074    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:06:32.756280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:32.756135    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 08:06:33.223251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:33.223196    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:33.754742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:33.754709    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:06:33.755216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:33.754760    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 08:06:34.265233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:34.265195    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:34.754359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:34.754328    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:06:34.754552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:34.754397    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 08:06:35.362414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:35.362383    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:35.754345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:35.754311    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:06:35.754516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:35.754372    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 08:06:36.267029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:36.266950    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:36.755222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:36.755188    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:06:36.755632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:36.755245    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 08:06:37.323821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:37.323789    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:37.754813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:37.754775    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:06:37.754985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:37.754844    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 08:06:38.263629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:38.263590    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:38.793632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:38.793604    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:38.794727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:38.794708    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:38.795303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:38.795283    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:06:39.055674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:39.055571    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:39.945400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:39.945355    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:40.671549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:40.671500    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:41.453793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:41.453753    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:42.236424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:42.236386    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:43.035340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:43.035309    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:43.853046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:43.853010    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:44.554518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:44.554488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:45.350319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.350285    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:45.773981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.773947    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"]
Nov 03 08:06:45.782494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.782467    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.802363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.802333    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"]
Nov 03 08:06:45.820626 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.820594    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"]
Nov 03 08:06:45.833459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.833433    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.841920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.841897    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"]
Nov 03 08:06:45.902907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.902824    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.902907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.902865    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.902907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.902901    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.903218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.902928    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870199b5-d3ba-473a-97e9-da52cfbeeee0-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.903218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.902980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903013    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9z59\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-kube-api-access-j9z59\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903042    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4423db23-dc99-4360-b61e-87b6b323abe5-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903089    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903127    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmz2\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-kube-api-access-qnmz2\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.903218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903168    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.903569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903244    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.903569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903276    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4423db23-dc99-4360-b61e-87b6b323abe5-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903384    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.903569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903423    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903478    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903510    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:45.903569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903559    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.903919 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.903605    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:45.913226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.913201    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"]
Nov 03 08:06:45.917380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.917363    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:45.937313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.937290    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"]
Nov 03 08:06:45.941076 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.941045    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:45.948679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.948653    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"]
Nov 03 08:06:45.973203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:45.973178    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"]
Nov 03 08:06:46.005270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005240    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.005401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005280    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9z59\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-kube-api-access-j9z59\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.005401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005309    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4423db23-dc99-4360-b61e-87b6b323abe5-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.005525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005504    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.005586 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005563    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmz2\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-kube-api-access-qnmz2\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.005649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005599    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.005695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005649    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.005763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005706    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.005828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005772    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.005828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005808    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.005955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005929    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.006016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.005940    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.006016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006001    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ftv\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-kube-api-access-z7ftv\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.006154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006037    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.006154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006089    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4423db23-dc99-4360-b61e-87b6b323abe5-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.006154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006119    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.006154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006147    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.006428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006157    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.006428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006294    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.006543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006432    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.006543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006477    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.006543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006513    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68451d0b-c751-4b74-a72c-2dcd31d07254-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.006710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006542    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.006710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006588    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.006710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006619    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.006710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006646    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.006710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006677    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.007029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006708    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49qp\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-kube-api-access-n49qp\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.007029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006743    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.007029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006772    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.007029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006815    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.007029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006816    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4423db23-dc99-4360-b61e-87b6b323abe5-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.007029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.006864    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.007029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007020    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.007447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007143    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.007447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007282    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.007447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007328    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.007447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007345    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.007447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007370    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.007708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007444    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.007708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007511    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.007708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007552    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.007708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007586    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.007708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.007616    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870199b5-d3ba-473a-97e9-da52cfbeeee0-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.008773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.008367    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870199b5-d3ba-473a-97e9-da52cfbeeee0-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.009092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.009021    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.009341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.009262    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4423db23-dc99-4360-b61e-87b6b323abe5-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.010159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.008267    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.010880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.010856    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.011022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.011000    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.022191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.022147    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.024622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.024569    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9z59\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-kube-api-access-j9z59\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.026621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.026598    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmz2\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-kube-api-access-qnmz2\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.027089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.027031    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.108749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108719    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.108943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108761    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.108943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108782    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.108943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108808    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.108943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ftv\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-kube-api-access-z7ftv\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.108943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108905    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.108943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108931    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.108969    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109032    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109134    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68451d0b-c751-4b74-a72c-2dcd31d07254-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109164    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109216    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109221    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109245    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109263    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109278    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49qp\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-kube-api-access-n49qp\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109485    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109514    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.109846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109750    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.109846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109792    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.109846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109813    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.109846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109817    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.110202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109948    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.110202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109960    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68451d0b-c751-4b74-a72c-2dcd31d07254-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.110202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.109977    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.110202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.110021    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.110202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.110128    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.111456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.111433    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.111850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.111827    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.112504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.112481    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.112667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.112650    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.113777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.113760    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.116349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.116326    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:46.117040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.116979    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.117398 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.117379    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ftv\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-kube-api-access-z7ftv\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.117802 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.117786    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.118318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.118288    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49qp\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-kube-api-access-n49qp\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.156337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.156304    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:46.231844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.231807    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:46.255753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.255476    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:46.301415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.301269    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:46.434685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.434650    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"]
Nov 03 08:06:46.459226 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:46.459161    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4423db23_dc99_4360_b61e_87b6b323abe5.slice/crio-ca768a3880bd3de1b8b72caaa0c0f34848aeeafd2d62fe2a064a49acb51d5664 WatchSource:0}: Error finding container ca768a3880bd3de1b8b72caaa0c0f34848aeeafd2d62fe2a064a49acb51d5664: Status 404 returned error can't find the container with id ca768a3880bd3de1b8b72caaa0c0f34848aeeafd2d62fe2a064a49acb51d5664
Nov 03 08:06:46.462274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.462232    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.462408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.462325    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.462408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.462366    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.477048 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.477023    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"]
Nov 03 08:06:46.480280 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:46.480258    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870199b5_d3ba_473a_97e9_da52cfbeeee0.slice/crio-47334e89ce95a7b49cbf47fbff05e0356af7ae2870d31b84987c6e22a94bd352 WatchSource:0}: Error finding container 47334e89ce95a7b49cbf47fbff05e0356af7ae2870d31b84987c6e22a94bd352: Status 404 returned error can't find the container with id 47334e89ce95a7b49cbf47fbff05e0356af7ae2870d31b84987c6e22a94bd352
Nov 03 08:06:46.483655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.483618    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.483803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.483699    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.483803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.483739    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.538148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.538116    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" event={"ID":"870199b5-d3ba-473a-97e9-da52cfbeeee0","Type":"ContainerStarted","Data":"47334e89ce95a7b49cbf47fbff05e0356af7ae2870d31b84987c6e22a94bd352"}
Nov 03 08:06:46.539794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.539765    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" event={"ID":"4423db23-dc99-4360-b61e-87b6b323abe5","Type":"ContainerStarted","Data":"ca768a3880bd3de1b8b72caaa0c0f34848aeeafd2d62fe2a064a49acb51d5664"}
Nov 03 08:06:46.575639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.575606    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"]
Nov 03 08:06:46.579343 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:46.579302    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907ee690_cf3d_43f1_8ece_6e7491fd8ad6.slice/crio-2f21fefe95712ed2fc820a30e8eddb254c20b90d5d9cdf4959edfba5058dff57 WatchSource:0}: Error finding container 2f21fefe95712ed2fc820a30e8eddb254c20b90d5d9cdf4959edfba5058dff57: Status 404 returned error can't find the container with id 2f21fefe95712ed2fc820a30e8eddb254c20b90d5d9cdf4959edfba5058dff57
Nov 03 08:06:46.589427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.588820    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.589427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.588886    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.589427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.588914    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.589609 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:06:46.589460    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68451d0b_c751_4b74_a72c_2dcd31d07254.slice/crio-a1d7a88523d0aed89a0be98689c755fb00fbfc0bcecd82492fbc67d4e8ac96b0 WatchSource:0}: Error finding container a1d7a88523d0aed89a0be98689c755fb00fbfc0bcecd82492fbc67d4e8ac96b0: Status 404 returned error can't find the container with id a1d7a88523d0aed89a0be98689c755fb00fbfc0bcecd82492fbc67d4e8ac96b0
Nov 03 08:06:46.595639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.595616    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"]
Nov 03 08:06:46.602568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.601867    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.602568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.601949    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:46.602568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:46.602002    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:06:47.589087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.588207    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" event={"ID":"68451d0b-c751-4b74-a72c-2dcd31d07254","Type":"ContainerStarted","Data":"0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580"}
Nov 03 08:06:47.589087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.588311    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" event={"ID":"68451d0b-c751-4b74-a72c-2dcd31d07254","Type":"ContainerStarted","Data":"a1d7a88523d0aed89a0be98689c755fb00fbfc0bcecd82492fbc67d4e8ac96b0"}
Nov 03 08:06:47.614991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.614923    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podStartSLOduration=2.6149024929999998 podStartE2EDuration="2.614902493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.612555638 +0000 UTC m=+757.115391150" watchObservedRunningTime="2026-11-03 08:06:47.614902493 +0000 UTC m=+757.117738010"
Nov 03 08:06:47.641273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.618568    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" event={"ID":"870199b5-d3ba-473a-97e9-da52cfbeeee0","Type":"ContainerStarted","Data":"9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb"}
Nov 03 08:06:47.671295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.671230    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podStartSLOduration=2.671208344 podStartE2EDuration="2.671208344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.67041946 +0000 UTC m=+757.173254972" watchObservedRunningTime="2026-11-03 08:06:47.671208344 +0000 UTC m=+757.174043873"
Nov 03 08:06:47.692702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.692651    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:47.694046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.694013    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" event={"ID":"4423db23-dc99-4360-b61e-87b6b323abe5","Type":"ContainerStarted","Data":"2d77e5ea66c12dfe20b916e99a8974b59170a17131414f5a1d0d12f582519758"}
Nov 03 08:06:47.737745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.737672    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podStartSLOduration=2.737652154 podStartE2EDuration="2.737652154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.736845153 +0000 UTC m=+757.239680660" watchObservedRunningTime="2026-11-03 08:06:47.737652154 +0000 UTC m=+757.240487669"
Nov 03 08:06:47.768998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.767824    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" event={"ID":"907ee690-cf3d-43f1-8ece-6e7491fd8ad6","Type":"ContainerStarted","Data":"95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec"}
Nov 03 08:06:47.768998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.767895    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" event={"ID":"907ee690-cf3d-43f1-8ece-6e7491fd8ad6","Type":"ContainerStarted","Data":"2f21fefe95712ed2fc820a30e8eddb254c20b90d5d9cdf4959edfba5058dff57"}
Nov 03 08:06:47.804082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:47.802115    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podStartSLOduration=2.802096684 podStartE2EDuration="2.802096684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.80138423 +0000 UTC m=+757.304219784" watchObservedRunningTime="2026-11-03 08:06:47.802096684 +0000 UTC m=+757.304932202"
Nov 03 08:06:48.123086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.122089    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:48.123086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.122287    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.123086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.122364    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:48.167086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.166251    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:48.180094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.179236    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.180094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.179333    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:48.239035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.239002    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:48.239424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.239401    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.239597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.239568    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:48.258268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.258239    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:48.258975 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.258944    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.291277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.291230    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:48.811311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.811269    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:48.813324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.813295    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:48.816526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.816500    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:48.863211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:48.863167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.094528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.093158    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.118613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.118576    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.118916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.118875    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:49.125809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.125768    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.158530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.158495    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.158776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.158745    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:49.237408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.237363    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.238279 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.238254    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.238545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.238520    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:49.264131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.264098    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.264378 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.264353    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:49.308583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.308548    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:49.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.788785    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:06:49.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.789560    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.819086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.812235    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.819086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.813017    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:49.819086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.813888    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.819086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.814458    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:06:49.823104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.822028    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:06:49.823104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.822360    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:06:49.866563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.866526    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.867504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:49.867481    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:06:50.035084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.031761    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.117142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.101904    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.117142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.102713    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:06:50.208084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.203504    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.208084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.203583    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:50.208084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.204280    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.208084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.204322    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:50.300088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.299589    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.300088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.299645    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:50.331092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.330856    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.331092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.330911    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:50.818096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.817369    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.878955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.878901    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 03 08:06:50.881190 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.881150    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.883113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.883086    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.884926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.884902    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.886252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.886229    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.914276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.906534    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.944093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.932809    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Nov 03 08:06:50.961569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:50.950684    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:06:51.133167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.127381    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:06:51.152944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.133624    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.182632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.172050    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.182632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.172139    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:51.240403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.240368    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.240660 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.240633    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:51.269915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.269882    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.275629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.270539    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:51.279956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.279930    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.280161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.280137    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:51.294788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.294763    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:06:51.304085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.304043    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:06:51.311913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.311882    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:06:51.312936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.312917    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:06:51.313141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.313128    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:06:51.429161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.387979    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:06:51.574102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.574036    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:06:51.886087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.884159    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.886087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.885316    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.899086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.897489    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.929085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.924159    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.966097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:51.952496    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.123094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.119483    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.123094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.119552    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:52.163602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.163502    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.163868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.163832    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:52.259114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.249192    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.259114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.249251    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:52.275154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.260230    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.275154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.260280    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:52.280177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.280138    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.620093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.618365    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:52.822093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.815272    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:06:52.897162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.888479    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.955087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.914552    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:06:52.955087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.931688    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.979088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.958186    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.981394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.979776    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:06:52.981394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:52.980161    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:06:53.119876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.119419    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.119876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.119480    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:53.191149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.159154    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.191149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.159775    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.191149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.159816    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:53.370091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.316288    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.370091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.316735    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:06:53.370091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.318686    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.370091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.318772    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:53.410088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.384759    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.476357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.476235    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:53.600089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.558604    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.838145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.825512    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:06:53.869598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.869568    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:06:53.902289 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.894362    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.924278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.923188    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.960510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.960468    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.977957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:53.968183    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:06:54.010087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.003888    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:54.098271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.098196    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:06:54.123147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.123116    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.123412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.123380    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:54.184290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.184166    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.184290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.184224    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:54.323087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.314623    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.323087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.314680    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:54.323087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.315331    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.323087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.315373    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:54.324128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.324104    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:06:54.369642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.369563    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:06:54.504087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.503764    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:06:54.557089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.556347    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:06:54.596106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.596055    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:06:54.701833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:54.701807    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:06:55.088499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.088416    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:55.120410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.119200    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.120410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.119261    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 08:06:55.165839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.165806    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.166130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.166099    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 08:06:55.239088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.235404    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.239088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.235466    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 08:06:55.261839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.257425    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.261839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:55.257476    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 08:06:56.126808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.126779    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:56.129110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.128995    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:56.129825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.129805    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:06:56.294086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.293844    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:56.299968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.298621    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:56.304344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.304304    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:06:56.399938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.399852    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:56.401870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.401737    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:56.403582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.403563    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:06:56.613560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.613532    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:56.672312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.672213    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:56.672688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.672668    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:56.674584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:56.674562    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:06:58.167870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:58.167836    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:06:58.377223 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:06:58.377174    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:06:59.211668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:06:59.211634    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:00.270386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:00.270349    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:01.373084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:01.372498    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.401089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:01.391488    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:01.562089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:01.561178    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.930091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:01.929590    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.969120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:01.967234    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.105265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:02.105200    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.572087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:02.569921    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:07:02.619218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:02.619181    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:03.013296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.013225    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:07:03.050089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.025362    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.075350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.059402    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:03.075553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.075408    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.078625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.078082    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.195867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.118691    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.195867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.153521    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.216467 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.216431    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:07:03.216799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.216770    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:07:03.222230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.222209    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:07:03.274508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.215372    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.412581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.412536    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.581340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:03.581235    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.032606 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.032556    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.082234 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.082185    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.102209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.102164    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.114182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.114140    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.126625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.126588    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.180087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.142249    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.180982 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.180953    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:07:04.181203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.181187    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:07:04.181357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.181345    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:07:04.393139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.373009    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.396268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.393162    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.396749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.396728    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:07:04.492715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.492678    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.493505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.493483    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:07:04.602247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.602203    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.656548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.656447    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:07:04.668078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.664046    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:07:04.700051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.700023    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:07:04.786228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.786202    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:07:04.786531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.786518    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:07:04.817140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:04.810452    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:07:05.053466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:05.053167    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.142924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:05.141848    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.180891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:05.180851    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.262129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:05.248529    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:07:05.291097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:05.283674    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:07:05.500685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:05.500654    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:06.166875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:06.164629    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:07.221023 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:07.220989    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:08.399287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:08.399251    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:09.145305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:09.145274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:10.121760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:10.121729    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:10.871055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:10.871026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:11.708788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:11.708755    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:11.832468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:11.832433    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:12.770488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:12.770449    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:13.701343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:13.701306    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:14.462130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:14.462100    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:15.262420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:15.262368    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:16.053891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:16.053856    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:16.931509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:16.931474    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:17.835209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:17.835177    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:18.651531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:18.651494    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:19.403901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:19.403842    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:20.442528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:20.442497    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:21.341875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:21.341838    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:22.425436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:22.425399    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:23.222854 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:23.222817    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:24.239354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:24.239324    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:25.195117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:25.195047    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:26.028534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:26.028496    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:27.506198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:27.506152    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:27.667905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:27.667863    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:27.802241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:27.802138    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:27.829086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:27.825210    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.045918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.045879    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.376603 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.376532    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:07:28.511610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.509162    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.511610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.510039    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:28.709801 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.709756    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.710750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.710720    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.827091 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:07:28.802463    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:07:28.841095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.829901    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:07:28.845223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.845189    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:28.880090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.853190    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.880090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.855280    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:28.930238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.930205    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:07:28.930614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:28.930594    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:07:29.106148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.105256    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.106148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.105724    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:07:29.132089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.107147    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:07:29.157087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.137151    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.207087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.206968    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.396263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.396165    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.397707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.397682    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:07:29.711543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.710197    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.741250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.740020    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:07:29.742149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.741563    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:07:29.744851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.742423    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:07:29.959135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.935470    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:29.998280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:29.959991    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.119212 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.119164    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.165526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.164567    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.165526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.165312    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:07:30.246133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.226136    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.246133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.228391    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:07:30.371202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.371086    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.409296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.404567    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.731242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.731195    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.790099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.773470    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.790099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.773788    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.803255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.803184    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:30.823536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.823481    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.872105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.852854    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.000150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.998178    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:31.000150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:30.998932    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.128088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.121822    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.131612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.131581    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:31.165251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.165213    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:31.230215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.230172    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.292140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.288554    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"]
Nov 03 08:07:31.378167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.378122    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.418088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.395568    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:07:31.418535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.418508    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"]
Nov 03 08:07:31.420118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.414499    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.421288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.414872    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.515123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.458128    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.515883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.515859    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"]
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566451    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqk5w\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-kube-api-access-hqk5w\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566496    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566538    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566598    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566639    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566710    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22dcaae-af57-4cd0-b304-fa20192b9a62-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566769    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566802    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566839    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566869    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566913    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566941    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.566990    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.572222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.567026    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.574188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.574144    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8prf\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-kube-api-access-d8prf\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.574308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.574225    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.574399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.574378    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.574455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.574418    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.680107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.671661    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682634    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682722    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682772    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22dcaae-af57-4cd0-b304-fa20192b9a62-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682899    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682931    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682960    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.682989    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683036    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683083    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683127    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683213    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8prf\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-kube-api-access-d8prf\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683259    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683368    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683400    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683445    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqk5w\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-kube-api-access-hqk5w\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.683475    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.684102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.684012    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.686319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.685813    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.725287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.721440    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.725287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.721727    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.725287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.722621    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22dcaae-af57-4cd0-b304-fa20192b9a62-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.725287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.722850    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.725287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.723117    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.732175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.732141    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"]
Nov 03 08:07:31.743577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.743518    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.744621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.744586    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.748937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.746444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.748937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.746739    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.799083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.789236    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.799083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.790919    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:07:31.844137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.823014    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8prf\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-kube-api-access-d8prf\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.845844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.839624    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.846390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.846366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqk5w\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-kube-api-access-hqk5w\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:31.888976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.888877    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:31.902185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.902142    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.030093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:31.949593    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:07:32.030093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.028425    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.030093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.029503    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:32.111146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.058052    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:32.137089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.111729    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:32.137089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.123224    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:07:32.207055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.207013    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.261119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.254616    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:32.264462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.264424    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:07:32.266023 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.265985    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:07:32.310834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.310633    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:32.495160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.455661    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.495160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.471153    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.513546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.513307    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:32.677987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:32.672755    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.048626 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.043555    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.048626 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.043938    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:07:33.128905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.127408    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.304838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.304643    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.307096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.307046    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.322797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.322756    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.391268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.391228    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:07:33.399699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.397983    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:07:33.458422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.457772    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:07:33.479719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.476326    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.546373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.544262    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:33.546373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.544500    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:33.546373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.544884    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:33.562820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.561272    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"]
Nov 03 08:07:33.569273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.569248    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:07:33.569767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.569750    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:07:33.587527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.579201    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:07:33.587527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.580637    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:07:33.598482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.598447    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:33.598688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.598669    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:33.598823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.598808    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:33.604955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.599489    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:07:33.613369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:33.612787    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"]
Nov 03 08:07:34.165087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.165025    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" event={"ID":"72d91edf-d0c3-42fa-85b5-7d76d72a562c","Type":"ContainerStarted","Data":"84ae452b147a5098d19318f68bf3787236331801de39235baab15fc8f96ba1f7"}
Nov 03 08:07:34.165518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.165118    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" event={"ID":"72d91edf-d0c3-42fa-85b5-7d76d72a562c","Type":"ContainerStarted","Data":"d2cf9a9c352b6e1fb5162c1b09fd1d77adfe9e3afeb281c8216dd14690327a09"}
Nov 03 08:07:34.166713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.166692    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" event={"ID":"a22dcaae-af57-4cd0-b304-fa20192b9a62","Type":"ContainerStarted","Data":"79d9fd01e53ecf240503776f1e6bc372d5f5f6c6db474a9bf19086258c49923f"}
Nov 03 08:07:34.166827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.166733    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" event={"ID":"a22dcaae-af57-4cd0-b304-fa20192b9a62","Type":"ContainerStarted","Data":"0767e028dcaf55e86cce179c4561a9432a09395ce1029c6e38bd8faa5798dfef"}
Nov 03 08:07:34.181786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.181742    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podStartSLOduration=4.181727352 podStartE2EDuration="4.181727352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:34.180366147 +0000 UTC m=+803.683201661" watchObservedRunningTime="2026-11-03 08:07:34.181727352 +0000 UTC m=+803.684562867"
Nov 03 08:07:34.198902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.198853    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podStartSLOduration=4.198841387 podStartE2EDuration="4.198841387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:34.196637889 +0000 UTC m=+803.699473406" watchObservedRunningTime="2026-11-03 08:07:34.198841387 +0000 UTC m=+803.701676978"
Nov 03 08:07:34.313484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.313449    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:34.313803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.313781    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 08:07:34.313953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.313930    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 08:07:34.522124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.521264    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:34.532088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.523345    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 08:07:34.532088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:34.523419    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 08:07:35.326211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:35.326175    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 08:07:35.326867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:35.326833    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 08:07:35.521089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:35.519608    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 08:07:35.521089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:35.519671    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 08:07:36.048086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:36.047515    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:36.316242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:36.315438    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 08:07:36.316242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:36.315501    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 08:07:36.528087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:36.521446    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 08:07:36.528087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:36.521523    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 08:07:37.173122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:37.173091    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:37.320090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:37.319643    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 08:07:37.320090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:37.319702    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 08:07:37.523088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:37.520458    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 08:07:37.523088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:37.520532    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 08:07:38.261745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:38.261712    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:38.329094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:38.327202    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 08:07:38.329094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:38.327283    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 08:07:38.517877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:38.517782    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 08:07:38.521391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:38.521353    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 08:07:39.321086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:39.319205    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 08:07:39.321086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:39.319278    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 08:07:39.332087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:39.325875    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:39.519831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:39.519796    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 08:07:39.528188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:39.528148    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 08:07:40.311814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:40.311779    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 08:07:40.311995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:40.311839    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 08:07:40.513873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:40.513830    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 08:07:40.514348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:40.513906    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 08:07:40.547935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:40.547901    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:41.503831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:41.503798    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:41.504669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:41.504643    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:41.505997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:41.505977    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:07:41.518970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:41.518949    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:41.519474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:41.519301    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:41.520828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:41.520809    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:07:41.724845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:41.724817    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:42.770335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:42.770308    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:43.613878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:43.613844    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:44.481810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:44.481777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:45.172856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:45.172823    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:46.210097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.191149    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"]
Nov 03 08:07:46.210097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.196604    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.218714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.218626    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"]
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.462690    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.462751    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.462785    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.462917    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.462966    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.463039    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2md\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-kube-api-access-rv2md\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.463151    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.463197    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.464115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.463278    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.522089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.506334    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.581990    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.582157    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.582196    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.582226    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.583037    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.583372    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.583422    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.583489    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2md\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-kube-api-access-rv2md\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.583579    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.583892    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.584178    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.584811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.590620    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.654093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.590701    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.756198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.755432    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.831252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.831217    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.871461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.871385    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:46.892909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:46.892840    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2md\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-kube-api-access-rv2md\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:47.122388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.121139    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:07:47.151724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.151386    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:47.434775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.432167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:47.434775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.432565    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:07:47.434775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.432861    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:47.434775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.433140    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded"
Nov 03 08:07:47.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.498045    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:47.601100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.582163    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:47.676380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.676322    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:47.801369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.801319    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:47.948122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:47.940360    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:07:48.025101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.012921    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:48.136087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.135125    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:48.136087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.135492    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:07:48.435110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.435041    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:48.449471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.435700    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:07:48.459480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.435733    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:48.475141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.458220    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:07:48.490674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.490647    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:07:48.491520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.491502    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:07:48.608135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.577183    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:48.608135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.595187    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:48.608135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.597201    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:07:48.608135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.597252    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:07:48.750482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.750439    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:48.750806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.750782    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:48.956090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.881894    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:48.956090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:48.950228    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.055573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.051500    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:07:49.055573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.051589    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:07:49.159085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.158387    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.159985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.159948    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.175231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.175196    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.183105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.180174    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:07:49.355471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.355173    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.422101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.412362    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:49.427098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.422493    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:49.427098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.422577    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:07:49.483499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.483444    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.485314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.485268    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.528435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.492357    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.632152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.622869    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.704567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.624407    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.706076 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.702359    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"]
Nov 03 08:07:49.765747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.764214    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.958087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.936888    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.958087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:49.951760    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.169087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.168191    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.235721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.214005    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.566204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.521143    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.566204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.524077    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.566204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.524418    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:50.631229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.631184    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.704352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.704305    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.732088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.726212    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:07:50.791089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.778847    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.868088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.846268    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Nov 03 08:07:50.941322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.941264    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.050099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:50.987801    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.302089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.162838    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" event={"ID":"f0510f54-3454-46df-b6ee-cbb7b53dfb8c","Type":"ContainerStarted","Data":"660a357c6573be2c511cb2151628b8632433bc6e8eb44ee298d74497a99757e8"}
Nov 03 08:07:51.316088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.213188    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.316088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.238929    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.526084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.523730    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.526084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.525167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.526084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.525486    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.723362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.723300    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.731311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.731266    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.829097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.780227    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.911111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.887761    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" event={"ID":"f0510f54-3454-46df-b6ee-cbb7b53dfb8c","Type":"ContainerStarted","Data":"c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d"}
Nov 03 08:07:51.911111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.895023    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.959490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:51.959440    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.182144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.161621    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:07:52.182144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.163222    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:52.182144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.163290    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:52.274093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.272279    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.329090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.325189    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.329090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.325981    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.543144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.529991    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.543144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.531476    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.543144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.531819    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.744085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.732134    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.828139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.827952    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.829093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.828306    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.965089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:52.964575    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:53.177147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.176328    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 08:07:53.177147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.177137    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:53.177385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.177177    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:53.307090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.286165    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:07:53.344086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.326587    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:53.344086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.328745    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:53.379420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.379389    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:07:53.421836 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.421732    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podStartSLOduration=7.421712351 podStartE2EDuration="7.421712351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:51.935722476 +0000 UTC m=+821.438558017" watchObservedRunningTime="2026-11-03 08:07:53.421712351 +0000 UTC m=+822.924547866"
Nov 03 08:07:53.615454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.615415    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:53.652121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.618719    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:07:53.652121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.619927    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:07:53.652121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.623549    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:07:53.747575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.747518    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:53.855479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:53.849823    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:07:54.067755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.067714    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:07:54.146648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.146598    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:07:54.181719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.181687    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:54.408678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.408576    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:54.409350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.244265    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:54.409541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.409517    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:54.410108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.410081    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:54.410934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.410911    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:07:54.616756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.616692    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:54.657467 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.627096    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:54.679694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.627983    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:54.755566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.755516    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:54.850684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.850647    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:54.864517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.864484    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:07:54.940966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.940937    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:07:54.942917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.942897    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:07:54.948767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.948748    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:07:54.949807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.949788    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:07:54.951573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.951555    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:07:54.951762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:54.951748    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:07:55.152072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:55.152021    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:55.152267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:55.152120    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:56.152134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:56.152100    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:56.152619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:56.152175    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:56.223309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:56.223273    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:57.152580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:57.152541    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:57.152982 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:57.152612    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:57.456705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:57.456668    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:58.060380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.060347    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6994-fb05bd"]
Nov 03 08:07:58.066196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.065984    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:07:58.068813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.068793    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfklg\""
Nov 03 08:07:58.069187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.069162    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6994-fb05bd"]
Nov 03 08:07:58.129089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.128921    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlg8c\" (UniqueName: \"kubernetes.io/projected/4acb5f2b-c32a-4aa2-bebf-e899db6c448b-kube-api-access-xlg8c\") pod \"test-curl-6994-fb05bd\" (UID: \"4acb5f2b-c32a-4aa2-bebf-e899db6c448b\") " pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:07:58.152412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.152381    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:58.152575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.152437    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:58.230112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.230040    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlg8c\" (UniqueName: \"kubernetes.io/projected/4acb5f2b-c32a-4aa2-bebf-e899db6c448b-kube-api-access-xlg8c\") pod \"test-curl-6994-fb05bd\" (UID: \"4acb5f2b-c32a-4aa2-bebf-e899db6c448b\") " pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:07:58.236944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.236915    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlg8c\" (UniqueName: \"kubernetes.io/projected/4acb5f2b-c32a-4aa2-bebf-e899db6c448b-kube-api-access-xlg8c\") pod \"test-curl-6994-fb05bd\" (UID: \"4acb5f2b-c32a-4aa2-bebf-e899db6c448b\") " pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:07:58.408222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.401303    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:07:58.528382 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:07:58.528311    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:07:58.796460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.796431    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6994-fb05bd"]
Nov 03 08:07:58.828324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:58.828300    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:07:59.150141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:59.149809    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:07:59.158084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:59.156087    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:07:59.158084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:59.156148    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:07:59.636305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:59.636267    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6994-fb05bd" event={"ID":"4acb5f2b-c32a-4aa2-bebf-e899db6c448b","Type":"ContainerStarted","Data":"3e348bf7ab53ef940f754700e959a50752c8e245d600b595445e04c8bee8f1da"}
Nov 03 08:07:59.909369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:07:59.909280    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:00.152340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:00.152306    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 08:08:00.152510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:00.152371    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 08:08:00.644913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:00.644877    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:01.225898 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.220741    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"]
Nov 03 08:08:01.241423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.241345    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.244776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.244736    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"]
Nov 03 08:08:01.330140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330106    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330162    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330316    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bea25b-da8a-4336-aeaf-f82b3e59d563-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330352    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330370    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330387    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330430    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330461    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72sxx\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-kube-api-access-72sxx\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.330736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.330485    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432240    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bea25b-da8a-4336-aeaf-f82b3e59d563-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432322    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432354    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432381    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432448    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432499    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72sxx\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-kube-api-access-72sxx\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432543    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432682    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.432749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.432726    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.433275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.433251    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.433959 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.433936    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bea25b-da8a-4336-aeaf-f82b3e59d563-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.434612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.434585    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.434905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.434878    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.435194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.435171    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.436814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.436793    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.437935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.437914    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.455305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.455264    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.457634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.457604    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72sxx\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-kube-api-access-72sxx\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.588264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.588136    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:08:01.588453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.588279    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:01.589346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.589321    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:08:01.590334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.590313    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:08:01.764112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.762236    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:01.967542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.964536    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"]
Nov 03 08:08:01.970379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.970337    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:08:01.970495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.970430    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:08:01.970495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:01.970472    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:08:02.662505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:02.662413    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6994-fb05bd" event={"ID":"4acb5f2b-c32a-4aa2-bebf-e899db6c448b","Type":"ContainerStarted","Data":"64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344"}
Nov 03 08:08:02.664246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:02.664209    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" event={"ID":"46bea25b-da8a-4336-aeaf-f82b3e59d563","Type":"ContainerStarted","Data":"9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34"}
Nov 03 08:08:02.664384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:02.664263    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" event={"ID":"46bea25b-da8a-4336-aeaf-f82b3e59d563","Type":"ContainerStarted","Data":"1b914ba5b2dcce2815654e61723f0a1bfd4fbd9e4d98b924d80ee79c2670fbd6"}
Nov 03 08:08:02.677271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:02.677229    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6994-fb05bd" podStartSLOduration=1.342581747 podStartE2EDuration="4.677215474s" totalImagesPullingTime="3.334633727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:02.673834425 +0000 UTC m=+832.176669942" watchObservedRunningTime="2026-11-03 08:08:02.677215474 +0000 UTC m=+832.180051016"
Nov 03 08:08:02.693731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:02.693686    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podStartSLOduration=1.693670391 podStartE2EDuration="1.693670391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:02.689816617 +0000 UTC m=+832.192652131" watchObservedRunningTime="2026-11-03 08:08:02.693670391 +0000 UTC m=+832.196505904"
Nov 03 08:08:02.733716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:02.733682    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:03.375851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.375812    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6994-fb05bd"]
Nov 03 08:08:03.379236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.379211    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6994-fb05bd"]
Nov 03 08:08:03.528107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.528025    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6994-4504b5"]
Nov 03 08:08:03.528805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.528774    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4acb5f2b-c32a-4aa2-bebf-e899db6c448b" containerName="test-curl-6994-fb05bd"
Nov 03 08:08:03.528805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.528803    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acb5f2b-c32a-4aa2-bebf-e899db6c448b" containerName="test-curl-6994-fb05bd"
Nov 03 08:08:03.528986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.528970    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acb5f2b-c32a-4aa2-bebf-e899db6c448b" containerName="test-curl-6994-fb05bd"
Nov 03 08:08:03.533142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.533120    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:03.538365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.538343    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6994-4504b5"]
Nov 03 08:08:03.555734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.555691    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:03.612819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.606791    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:03.634160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.625601    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 08:08:03.634160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.625660    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 08:08:03.672122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.659965    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tb8\" (UniqueName: \"kubernetes.io/projected/776a5e30-ee13-4498-8979-6bd022de3dc6-kube-api-access-b8tb8\") pod \"test-curl-6994-4504b5\" (UID: \"776a5e30-ee13-4498-8979-6bd022de3dc6\") " pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:03.762763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.762694    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tb8\" (UniqueName: \"kubernetes.io/projected/776a5e30-ee13-4498-8979-6bd022de3dc6-kube-api-access-b8tb8\") pod \"test-curl-6994-4504b5\" (UID: \"776a5e30-ee13-4498-8979-6bd022de3dc6\") " pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:03.772966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.772938    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tb8\" (UniqueName: \"kubernetes.io/projected/776a5e30-ee13-4498-8979-6bd022de3dc6-kube-api-access-b8tb8\") pod \"test-curl-6994-4504b5\" (UID: \"776a5e30-ee13-4498-8979-6bd022de3dc6\") " pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:03.850620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:03.850583    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:04.156589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.156561    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6994-4504b5"]
Nov 03 08:08:04.158401 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:08:04.158377    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod776a5e30_ee13_4498_8979_6bd022de3dc6.slice/crio-3bf82821b0f55c3f54f2563dce2ee062760f44eef47ca7411c0e35904c233138 WatchSource:0}: Error finding container 3bf82821b0f55c3f54f2563dce2ee062760f44eef47ca7411c0e35904c233138: Status 404 returned error can't find the container with id 3bf82821b0f55c3f54f2563dce2ee062760f44eef47ca7411c0e35904c233138
Nov 03 08:08:04.547442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.547355    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:04.595151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.593459    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 08:08:04.595151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.593524    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 08:08:04.781227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.781193    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6994-4504b5" event={"ID":"776a5e30-ee13-4498-8979-6bd022de3dc6","Type":"ContainerStarted","Data":"d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5"}
Nov 03 08:08:04.781424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.781237    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6994-4504b5" event={"ID":"776a5e30-ee13-4498-8979-6bd022de3dc6","Type":"ContainerStarted","Data":"3bf82821b0f55c3f54f2563dce2ee062760f44eef47ca7411c0e35904c233138"}
Nov 03 08:08:04.781424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.781324    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6994-fb05bd" podUID="4acb5f2b-c32a-4aa2-bebf-e899db6c448b" containerName="test-curl-6994-fb05bd" containerID="cri-o://64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344" gracePeriod=2
Nov 03 08:08:04.795054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:04.795007    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6994-4504b5" podStartSLOduration=1.794993969 podStartE2EDuration="1.794993969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:04.792811698 +0000 UTC m=+834.295647207" watchObservedRunningTime="2026-11-03 08:08:04.794993969 +0000 UTC m=+834.297829485"
Nov 03 08:08:05.070939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.070911    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:05.204833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.204797    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:05.533853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.531296    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6994-4504b5"]
Nov 03 08:08:05.535877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.535849    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6994-4504b5"]
Nov 03 08:08:05.589387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.589344    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 08:08:05.589771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.589421    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 08:08:05.643039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.643010    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:05.756085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:05.755694    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:06.328255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.328207    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:06.441523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.441494    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:06.540492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.540459    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:06.596135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.593989    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 08:08:06.596135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.594054    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 08:08:06.840381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.839860    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6994-4504b5" podUID="776a5e30-ee13-4498-8979-6bd022de3dc6" containerName="test-curl-6994-4504b5" containerID="cri-o://d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5" gracePeriod=2
Nov 03 08:08:06.880474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.880443    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:06.995833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:06.995796    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:07.118152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.113621    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:07.441537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.441450    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:07.577088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.569573    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:08:07.594434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.592533    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 08:08:07.594434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.592596    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 08:08:07.794146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.793085    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acb5f2b-c32a-4aa2-bebf-e899db6c448b-kube-api-access-xlg8c\" (UniqueName: \"kubernetes.io/projected/4acb5f2b-c32a-4aa2-bebf-e899db6c448b-kube-api-access-xlg8c\") pod \"4acb5f2b-c32a-4aa2-bebf-e899db6c448b\" (UID: \"4acb5f2b-c32a-4aa2-bebf-e899db6c448b\") "
Nov 03 08:08:07.831266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.831217    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acb5f2b-c32a-4aa2-bebf-e899db6c448b-kube-api-access-xlg8c" pod "4acb5f2b-c32a-4aa2-bebf-e899db6c448b" (UID: "4acb5f2b-c32a-4aa2-bebf-e899db6c448b"). InnerVolumeSpecName "kube-api-access-xlg8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:07.849199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.849167    6430 generic.go:356] "Generic (PLEG): container finished" podID="4acb5f2b-c32a-4aa2-bebf-e899db6c448b" containerID="64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344" exitCode=137
Nov 03 08:08:07.849351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.849313    6430 scope.go:122] "RemoveContainer" containerID="64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344"
Nov 03 08:08:07.849431 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.849419    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:08:07.852977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.852947    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:07.859516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.859499    6430 scope.go:122] "RemoveContainer" containerID="64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344"
Nov 03 08:08:07.859822 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:07.859802    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344\": container with ID starting with 64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344 not found: ID does not exist" containerID="64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344"
Nov 03 08:08:07.859892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.859830    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344"} err="failed to get container status \"64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344\": rpc error: code = NotFound desc = could not find container \"64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344\": container with ID starting with 64d8d21ade30917cb1dcc97cd6ca39d40216ecf5a1e83b01c0cd1747a1c19344 not found: ID does not exist"
Nov 03 08:08:07.861187 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:07.861163    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6994-fb05bd\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="4acb5f2b-c32a-4aa2-bebf-e899db6c448b" pod="opendatahub/test-curl-6994-fb05bd"
Nov 03 08:08:07.903452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:07.903397    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlg8c\" (UniqueName: \"kubernetes.io/projected/4acb5f2b-c32a-4aa2-bebf-e899db6c448b-kube-api-access-xlg8c\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:08.589418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:08.589383    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 08:08:08.589606 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:08.589442    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 08:08:08.621014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:08.620976    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4acb5f2b-c32a-4aa2-bebf-e899db6c448b" path="/var/lib/kubelet/pods/4acb5f2b-c32a-4aa2-bebf-e899db6c448b/volumes"
Nov 03 08:08:08.938600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:08.938563    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:09.206227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.206195    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:09.269678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.269642    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776a5e30-ee13-4498-8979-6bd022de3dc6-kube-api-access-b8tb8\" (UniqueName: \"kubernetes.io/projected/776a5e30-ee13-4498-8979-6bd022de3dc6-kube-api-access-b8tb8\") pod \"776a5e30-ee13-4498-8979-6bd022de3dc6\" (UID: \"776a5e30-ee13-4498-8979-6bd022de3dc6\") "
Nov 03 08:08:09.301316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.301279    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776a5e30-ee13-4498-8979-6bd022de3dc6-kube-api-access-b8tb8" pod "776a5e30-ee13-4498-8979-6bd022de3dc6" (UID: "776a5e30-ee13-4498-8979-6bd022de3dc6"). InnerVolumeSpecName "kube-api-access-b8tb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:09.387764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.387729    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tb8\" (UniqueName: \"kubernetes.io/projected/776a5e30-ee13-4498-8979-6bd022de3dc6-kube-api-access-b8tb8\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:09.589302 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.589260    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 08:08:09.589495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.589331    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 08:08:09.872934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.872843    6430 generic.go:356] "Generic (PLEG): container finished" podID="776a5e30-ee13-4498-8979-6bd022de3dc6" containerID="d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5" exitCode=137
Nov 03 08:08:09.873239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.873024    6430 scope.go:122] "RemoveContainer" containerID="d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5"
Nov 03 08:08:09.873239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.873182    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:09.886782 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:09.886689    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6994-4504b5\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="776a5e30-ee13-4498-8979-6bd022de3dc6" pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:09.887025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.887002    6430 scope.go:122] "RemoveContainer" containerID="d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5"
Nov 03 08:08:09.887408 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:09.887379    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5\": container with ID starting with d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5 not found: ID does not exist" containerID="d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5"
Nov 03 08:08:09.887493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:09.887421    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5"} err="failed to get container status \"d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5\": rpc error: code = NotFound desc = could not find container \"d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5\": container with ID starting with d9b12825b0a0b8a588b8356ac312cfdfcbdbb60b9a975303e9926543e2bd0fa5 not found: ID does not exist"
Nov 03 08:08:10.013196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.013165    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:10.263272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.207582    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:10.317087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.314473    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:10.594372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.594288    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:10.595456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.595435    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:10.596268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.596242    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:08:10.601456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.601432    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:10.620338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.620310    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776a5e30-ee13-4498-8979-6bd022de3dc6" path="/var/lib/kubelet/pods/776a5e30-ee13-4498-8979-6bd022de3dc6/volumes"
Nov 03 08:08:10.654822 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:10.654783    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6994-4504b5\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="776a5e30-ee13-4498-8979-6bd022de3dc6" pod="opendatahub/test-curl-6994-4504b5"
Nov 03 08:08:10.789987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.789957    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:10.934987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:10.934953    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:11.049851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:11.049818    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:11.327717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:11.327624    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:12.111295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.111261    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:12.232087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.228899    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:12.377138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.367567    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:12.491123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.486984    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:12.552994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.552937    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:12.681144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.680351    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:12.763212 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.763165    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:12.818890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.818855    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:12.871500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.871450    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:12.959299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.959205    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:12.997239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:12.997195    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.167711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.161890    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"]
Nov 03 08:08:13.169355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.169323    6430 cpu_manager.go:414] "containerMap: removing container" podUID="776a5e30-ee13-4498-8979-6bd022de3dc6" containerName="test-curl-6994-4504b5"
Nov 03 08:08:13.169522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.169505    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776a5e30-ee13-4498-8979-6bd022de3dc6" containerName="test-curl-6994-4504b5"
Nov 03 08:08:13.181086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.169912    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="776a5e30-ee13-4498-8979-6bd022de3dc6" containerName="test-curl-6994-4504b5"
Nov 03 08:08:13.183288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.183253    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.214881 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.214802    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"]
Nov 03 08:08:13.215260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.215184    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.253099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.250725    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.311397    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0b397f45-gw-openshift-default-dockercfg-fvpg7\""
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312354    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-certs\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312409    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-data\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312564    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-socket\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312600    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istiod-ca-cert\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312632    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-credential-socket\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312694    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-podinfo\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312757    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsgfz\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-kube-api-access-hsgfz\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312787    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-token\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.313085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.312831    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-envoy\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.359580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.347504    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.393084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.391516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416042    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-certs\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416142    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-data\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416352    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-socket\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416395    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istiod-ca-cert\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416432    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-credential-socket\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416507    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-podinfo\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416577    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgfz\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-kube-api-access-hsgfz\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416607    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-token\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.416654    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-envoy\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.417815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.417631    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istiod-ca-cert\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.432088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.417918    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-certs\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.432720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.432692    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-socket\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.506940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.442124    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-data\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.506940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.443029    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-credential-socket\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.508947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.508908    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.513085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.509678    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:08:13.560392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.560359    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:13.629086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.622105    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.629086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.598794    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-token\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.660474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.660409    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-podinfo\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:13.773221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.764126    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.773221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.764590    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:08:13.773221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.772663    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.958090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.901663    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.069210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.966254    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:08:14.069210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.967232    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.069210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.967571    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:14.069210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:13.967900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:14.069210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.027593    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.123091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.096373    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgfz\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-kube-api-access-hsgfz\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:14.224090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.159800    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:08:14.262385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.262339    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.273088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.264801    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:08:14.337250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.323348    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-envoy\") pod \"e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") " pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:14.349086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.348893    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.357160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.349950    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:08:14.552094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.547595    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:14.659430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.554170    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.685724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.608178    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.685724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.623242    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.775225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.775182    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.775826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.775793    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.822106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.790132    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:08:14.822106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.790198    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:08:14.951688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.951613    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Nov 03 08:08:14.985583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.985525    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.986514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:14.986480    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.183102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.159470    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.209141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.167990    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.319086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.286680    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.361088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.355190    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Nov 03 08:08:15.361088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.357983    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.720090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.704383    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.756089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.720451    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.756089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.720493    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.756089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.722239    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:08:15.778193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.778145    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.809089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:15.799742    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.046524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.022257    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.251684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.251628    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.260869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.260822    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.618089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.606592    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:16.619563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.619516    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.645880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.627642    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.737088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.725167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.737088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.725561    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.737088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.725843    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.815951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.815911    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.816537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.816505    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.903322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:16.903230    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:17.255747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.255700    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:17.331614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.331581    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:08:17.333378 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.333352    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:17.453086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.451372    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:17.465613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.465584    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"]
Nov 03 08:08:17.491087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.475616    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:17.491087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.475756    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.491087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.476105    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded"
Nov 03 08:08:17.503095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.499348    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:17.553450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.553367    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c855f06a\""
Nov 03 08:08:17.575583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.568274    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c855f06a-dockercfg-nk4k9\""
Nov 03 08:08:17.644084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.643192    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:17.644084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.643266    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:17.683573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.677147    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:17.716313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.713210    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"]
Nov 03 08:08:17.815144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.805129    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0084481-266e-4a9c-99a2-e6e9320d9712-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:17.815144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.805469    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkkg\" (UniqueName: \"kubernetes.io/projected/a0084481-266e-4a9c-99a2-e6e9320d9712-kube-api-access-9qkkg\") pod \"payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:17.884252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.870401    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:08:17.904210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.900800    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.904210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.902133    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.935090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.922700    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0084481-266e-4a9c-99a2-e6e9320d9712-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:17.935090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.922969    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkkg\" (UniqueName: \"kubernetes.io/projected/a0084481-266e-4a9c-99a2-e6e9320d9712-kube-api-access-9qkkg\") pod \"payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:17.935090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.926674    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0084481-266e-4a9c-99a2-e6e9320d9712-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:17.976088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:17.964602    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:08:18.021443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.005211    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.075662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.067764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkkg\" (UniqueName: \"kubernetes.io/projected/a0084481-266e-4a9c-99a2-e6e9320d9712-kube-api-access-9qkkg\") pod \"payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:18.093193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.093158    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:18.126345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.126309    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:18.262866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.262837    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:08:18.263156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.263135    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"]
Nov 03 08:08:18.265748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.128466    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.565564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.565519    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:08:18.565806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.565782    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:08:18.565950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.565930    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:08:18.757092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.713888    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.905724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.905690    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:08:18.923157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:18.923041    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:08:19.037670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.030709    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:19.066313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.066244    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.229090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.227168    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Nov 03 08:08:19.257100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.249638    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.260301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.260184    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded"
Nov 03 08:08:19.260301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.260228    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.282315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.269194    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.369710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.369668    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424" event={"ID":"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e","Type":"ContainerStarted","Data":"ee44b1ff6598c39a367f72feeeef1cc63a6d4ee9604b327adaf394246099190e"}
Nov 03 08:08:19.760107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.759149    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.858090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.854150    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.921255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.921200    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.984126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:19.978507    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.041490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.040164    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:20.041490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.040551    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:08:20.134331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.070525    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.134331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.071093    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:08:20.276122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.260279    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:08:20.276122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.261675    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.276490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.276464    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:20.277813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.277780    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.645094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.641543    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.791087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:20.778171    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:21.083093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.080171    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:08:21.131120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.099642    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:08:21.131567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.131520    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:21.194038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.166036    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:21.194038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.166386    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:21.194038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.177879    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:08:21.335446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.274212    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:08:21.335446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.279158    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:21.335446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.295641    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:08:21.370661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.366165    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424" event={"ID":"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e","Type":"ContainerStarted","Data":"00310e7fe784cd3c2ab322e6671d28ddb0e912f04454101d458ab49fcf1e6704"}
Nov 03 08:08:21.373592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.370903    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"]
Nov 03 08:08:21.373592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.371034    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:08:21.373592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.371097    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:08:21.398785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.398719    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424" podStartSLOduration=8.39868462 podStartE2EDuration="8.39868462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:21.375743028 +0000 UTC m=+850.878578543" watchObservedRunningTime="2026-11-03 08:08:21.39868462 +0000 UTC m=+850.901520135"
Nov 03 08:08:21.578536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.578505    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:21.679909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.679809    6430 patch_prober.go:29] interesting pod/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 08:08:21.688270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.688221    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424" podUID="098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 08:08:21.751553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.709213    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:08:21.784007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.783971    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:21.880827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.880776    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:21.976284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:21.976258    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:08:22.021843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.021797    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:22.091084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.087149    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:22.102348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.101112    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:22.151042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.151012    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:22.151454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.151438    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:08:22.165087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.158237    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:22.165087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.159360    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:22.181086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.178261    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:22.223421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.223392    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:08:22.284182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.284086    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:08:22.291512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.291484    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:08:22.337400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.337367    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c" event={"ID":"a0084481-266e-4a9c-99a2-e6e9320d9712","Type":"ContainerStarted","Data":"b6b9cf4466eb2935e7e321bb829fd6637ceffe8907e0699b6143196e220714c8"}
Nov 03 08:08:22.343336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.343271    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:08:22.715796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.715744    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:22.851721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.850172    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:22.851721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:22.850250    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:08:23.062166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:23.061212    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:08:23.095962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:23.095928    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:08:23.160986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:23.160931    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:23.298216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:23.298180    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:23.348927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:23.348743    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:23.349951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:23.349927    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:08:23.894108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:23.894042    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:24.036086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:24.028709    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:24.491574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:24.491534    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:24.694089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:24.691927    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:25.495514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:25.495443    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:26.370910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:26.370746    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c" event={"ID":"a0084481-266e-4a9c-99a2-e6e9320d9712","Type":"ContainerStarted","Data":"922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281"}
Nov 03 08:08:26.370910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:26.370834    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:26.384536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:26.384483    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c" podStartSLOduration=4.836043735 podStartE2EDuration="9.384470908s" totalImagesPullingTime="4.548427173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:26.38378727 +0000 UTC m=+855.886622786" watchObservedRunningTime="2026-11-03 08:08:26.384470908 +0000 UTC m=+855.887306421"
Nov 03 08:08:26.671965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:26.671875    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:27.316624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:27.316588    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:27.442219 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:27.442188    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:27.582176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:27.581037    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:27.700309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:27.700274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:28.331833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:28.331793    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:28.492812 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:28.492781    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:08:29.367176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:29.351903    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:30.165830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.165799    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"]
Nov 03 08:08:30.170050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.170028    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.175772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.175748    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0a9dd59a-serving-cert\""
Nov 03 08:08:30.179770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.179748    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"]
Nov 03 08:08:30.235314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.235267    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9c9347-5cd9-4919-9108-f5d40a06bd83-openshift-service-ca\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.235546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.235516    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.235727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.235617    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-metrics-certs\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.235727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.235670    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgqh\" (UniqueName: \"kubernetes.io/projected/fb9c9347-5cd9-4919-9108-f5d40a06bd83-kube-api-access-8lgqh\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.309018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.308984    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:30.336801 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.336752    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-metrics-certs\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.336983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.336824    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgqh\" (UniqueName: \"kubernetes.io/projected/fb9c9347-5cd9-4919-9108-f5d40a06bd83-kube-api-access-8lgqh\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.337047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.336980    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9c9347-5cd9-4919-9108-f5d40a06bd83-openshift-service-ca\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.337212 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.337189    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.337368 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:30.337315    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-serving-cert: secret "maas-api-e2e-ait-0a9dd59a-serving-cert" not found
Nov 03 08:08:30.337444 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:30.337396    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls podName:fb9c9347-5cd9-4919-9108-f5d40a06bd83 nodeName:}" failed. No retries permitted until 2026-11-03 08:08:30.837374825 +0000 UTC m=+860.340210339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls") pod "maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj" (UID: "fb9c9347-5cd9-4919-9108-f5d40a06bd83") : secret "maas-api-e2e-ait-0a9dd59a-serving-cert" not found
Nov 03 08:08:30.337702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.337679    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9c9347-5cd9-4919-9108-f5d40a06bd83-openshift-service-ca\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.361125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.339691    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-metrics-certs\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.385007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.370178    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgqh\" (UniqueName: \"kubernetes.io/projected/fb9c9347-5cd9-4919-9108-f5d40a06bd83-kube-api-access-8lgqh\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.449369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.449310    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:30.592501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.592466    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:30.860132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.856082    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:30.890085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:30.884090    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls\") pod \"maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:31.102488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.091242    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:31.194637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.194614    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"]
Nov 03 08:08:31.206417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.205116    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"]
Nov 03 08:08:31.206417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.205193    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.230977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.229317    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c855f06a-serving-cert\""
Nov 03 08:08:31.277275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.277220    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-metrics-certs\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.277544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.277516    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60c792b-e425-4583-9cb9-155678a50975-openshift-service-ca\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.277704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.277684    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dr8\" (UniqueName: \"kubernetes.io/projected/a60c792b-e425-4583-9cb9-155678a50975-kube-api-access-h2dr8\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.277785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.277757    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-maas-api-tls\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.278532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.278509    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:31.379012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.378931    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dr8\" (UniqueName: \"kubernetes.io/projected/a60c792b-e425-4583-9cb9-155678a50975-kube-api-access-h2dr8\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.379012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.378996    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-maas-api-tls\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.379346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.379102    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-metrics-certs\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.379346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.379302    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60c792b-e425-4583-9cb9-155678a50975-openshift-service-ca\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.379945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.379926    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60c792b-e425-4583-9cb9-155678a50975-openshift-service-ca\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.381732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.381699    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-maas-api-tls\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.381732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.381713    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-metrics-certs\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.404095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.387929    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dr8\" (UniqueName: \"kubernetes.io/projected/a60c792b-e425-4583-9cb9-155678a50975-kube-api-access-h2dr8\") pod \"maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.430104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.420814    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:31.529156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.514515    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"]
Nov 03 08:08:31.556099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.538205    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:31.585625 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:08:31.581540    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9c9347_5cd9_4919_9108_f5d40a06bd83.slice/crio-8fa318711e77cf501c5799d95f7394999290b6389fce7cdf62b2145810274c90 WatchSource:0}: Error finding container 8fa318711e77cf501c5799d95f7394999290b6389fce7cdf62b2145810274c90: Status 404 returned error can't find the container with id 8fa318711e77cf501c5799d95f7394999290b6389fce7cdf62b2145810274c90
Nov 03 08:08:31.872926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:31.872886    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"]
Nov 03 08:08:31.873850 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:08:31.873822    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60c792b_e425_4583_9cb9_155678a50975.slice/crio-dc6f30801c38c3f27ebfbee3894e39cb33893e6885ca984fd762b5348060216f WatchSource:0}: Error finding container dc6f30801c38c3f27ebfbee3894e39cb33893e6885ca984fd762b5348060216f: Status 404 returned error can't find the container with id dc6f30801c38c3f27ebfbee3894e39cb33893e6885ca984fd762b5348060216f
Nov 03 08:08:32.348250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:32.348220    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:32.521199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:32.521099    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj" event={"ID":"fb9c9347-5cd9-4919-9108-f5d40a06bd83","Type":"ContainerStarted","Data":"a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401"}
Nov 03 08:08:32.521199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:32.521164    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj" event={"ID":"fb9c9347-5cd9-4919-9108-f5d40a06bd83","Type":"ContainerStarted","Data":"8fa318711e77cf501c5799d95f7394999290b6389fce7cdf62b2145810274c90"}
Nov 03 08:08:32.521199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:32.521195    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:32.522668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:32.522644    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd" event={"ID":"a60c792b-e425-4583-9cb9-155678a50975","Type":"ContainerStarted","Data":"dc6f30801c38c3f27ebfbee3894e39cb33893e6885ca984fd762b5348060216f"}
Nov 03 08:08:32.547578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:32.547526    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj" podStartSLOduration=1.946637623 podStartE2EDuration="2.547512685s" totalImagesPullingTime="600.875062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:32.545423439 +0000 UTC m=+862.048258953" watchObservedRunningTime="2026-11-03 08:08:32.547512685 +0000 UTC m=+862.050348198"
Nov 03 08:08:33.146800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.146760    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"]
Nov 03 08:08:33.154703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.154672    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.174301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.174149    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0b397f45-serving-cert\""
Nov 03 08:08:33.175527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.175486    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"]
Nov 03 08:08:33.227542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.227508    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:33.235104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.235075    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50008583-2f67-4f0a-aa77-763cff2726bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.235381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.235363    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9n5\" (UniqueName: \"kubernetes.io/projected/50008583-2f67-4f0a-aa77-763cff2726bc-kube-api-access-7f9n5\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.235466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.235394    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-metrics-certs\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.235466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.235430    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.340083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.337190    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.340083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.337362    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50008583-2f67-4f0a-aa77-763cff2726bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.340083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.337624    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9n5\" (UniqueName: \"kubernetes.io/projected/50008583-2f67-4f0a-aa77-763cff2726bc-kube-api-access-7f9n5\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.340083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.337654    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-metrics-certs\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.352745 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:33.351822    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-serving-cert: secret "maas-api-e2e-migrate-0b397f45-serving-cert" not found
Nov 03 08:08:33.352745 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:33.351904    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls podName:50008583-2f67-4f0a-aa77-763cff2726bc nodeName:}" failed. No retries permitted until 2026-11-03 08:08:33.851881756 +0000 UTC m=+863.354717252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls") pod "maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w" (UID: "50008583-2f67-4f0a-aa77-763cff2726bc") : secret "maas-api-e2e-migrate-0b397f45-serving-cert" not found
Nov 03 08:08:33.353231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.353167    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50008583-2f67-4f0a-aa77-763cff2726bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.353368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.353344    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:33.356469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.356444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-metrics-certs\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.367008 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.366356    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9n5\" (UniqueName: \"kubernetes.io/projected/50008583-2f67-4f0a-aa77-763cff2726bc-kube-api-access-7f9n5\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.479157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.479115    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:33.530440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.530406    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd" event={"ID":"a60c792b-e425-4583-9cb9-155678a50975","Type":"ContainerStarted","Data":"17cc3b3ac6a4ec7d436e9cc86d785025f08ace6b48d434d6aa80c4a8b1d34821"}
Nov 03 08:08:33.530855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.530651    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:33.556888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.556837    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd" podStartSLOduration=1.859508377 podStartE2EDuration="2.556824414s" totalImagesPullingTime="697.316037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:33.556672982 +0000 UTC m=+863.059508496" watchObservedRunningTime="2026-11-03 08:08:33.556824414 +0000 UTC m=+863.059659927"
Nov 03 08:08:33.705147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.705116    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:33.832500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.832413    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:33.955565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.955313    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:33.981111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:33.981009    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls\") pod \"maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:34.034108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:34.030900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:34.088194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:34.085337    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:34.569147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:34.566154    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:34.882888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:34.882859    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"]
Nov 03 08:08:34.884814 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:08:34.884785    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50008583_2f67_4f0a_aa77_763cff2726bc.slice/crio-22deb319e15ba58ba3ff88810001aeae41951a3d82c5b97d410f55e906bbb847 WatchSource:0}: Error finding container 22deb319e15ba58ba3ff88810001aeae41951a3d82c5b97d410f55e906bbb847: Status 404 returned error can't find the container with id 22deb319e15ba58ba3ff88810001aeae41951a3d82c5b97d410f55e906bbb847
Nov 03 08:08:35.507943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:35.507900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:35.593332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:35.593290    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w" event={"ID":"50008583-2f67-4f0a-aa77-763cff2726bc","Type":"ContainerStarted","Data":"e1adbfb90d116b9caeadb415c4ebf106e1b9e8e56378e6c07673079747124ab6"}
Nov 03 08:08:35.593925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:35.593897    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w" event={"ID":"50008583-2f67-4f0a-aa77-763cff2726bc","Type":"ContainerStarted","Data":"22deb319e15ba58ba3ff88810001aeae41951a3d82c5b97d410f55e906bbb847"}
Nov 03 08:08:35.599227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:35.595305    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:35.624085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:35.618157    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w" podStartSLOduration=2.11412836 podStartE2EDuration="2.618135628s" totalImagesPullingTime="504.007268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:35.61648078 +0000 UTC m=+865.119316294" watchObservedRunningTime="2026-11-03 08:08:35.618135628 +0000 UTC m=+865.120971147"
Nov 03 08:08:35.624085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:35.622453    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:36.260334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:36.260274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:37.060492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:37.060458    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:37.380482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:37.380407    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:08:37.691929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:37.691903    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:37.856648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:37.856616    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:38.535757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:38.535727    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:08:38.632362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:38.632331    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:39.409844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:39.409811    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:39.661373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:39.661266    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:08:40.257332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:40.257299    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:40.449269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:40.449237    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:40.650383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:40.650297    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:41.331163 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:41.331132    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:41.498408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:41.498355    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:42.594052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:42.593991    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:42.648146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:42.648112    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:08:43.445171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:43.445127    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:43.794704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:43.794601    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:44.258560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:44.258526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:45.138095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:45.138019    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:45.863238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:45.863206    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:46.903242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:46.903208    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:49.232258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:49.232206    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.289086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:49.249181    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.323088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:49.319097    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.363225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:49.363172    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.392086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:49.377905    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.696373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:49.696329    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.920094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:49.918369    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.012128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.005333    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:08:50.012128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.011413    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.055086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.029437    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:08:50.298544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.298439    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.300330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.300304    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:08:50.357725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.342949    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.357725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.350830    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:08:50.357725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.351172    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:08:50.410610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.369204    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.410610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.381194    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:50.410610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.378077    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:08:50.410951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.410680    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:08:50.748465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.748416    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.793088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.789561    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.920087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.917506    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.947088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:50.928262    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.007673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.007560    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.012302 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.012266    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.114094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.110494    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:08:51.114094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.110558    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:08:51.114094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.110607    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:08:51.114094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.110648    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:08:51.129201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.129153    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.139088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.135106    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:08:51.306015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.305903    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.371091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.367520    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.371091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.367901    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.371091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.368515    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:08:51.371091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.369465    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.434097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.427142    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.818277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.818234    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.918154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.918108    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.973943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:51.959150    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.017266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.011715    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.017266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.013441    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.134145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.133168    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.319224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.319174    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.497176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.497128    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.517096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.500569    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.525087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.500601    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.525479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.500622    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.596942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.596903    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:08:52.876139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.851952    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.962218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.962168    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.999089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:52.995971    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:53.031154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.031119    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:08:53.046286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.045601    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.062576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.061410    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:53.181208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.181159    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.216104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.212969    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 08:08:53.326216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.326164    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.523244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.508117    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:08:53.548102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.524282    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:08:53.870145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.869244    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.970100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.970038    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.995085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.984198    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bm2g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:53.995085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:53.984284    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.037218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.037168    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.062512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.062470    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:08:54.185284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.185235    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.329443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.329396    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.377191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.377143    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:08:54.510239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.510139    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:08:54.527188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.527144    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.631771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.630812    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.727546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.727171    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:54.727546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.727249    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.864632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.864231    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:54.864632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.864297    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.889096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.888171    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:54.898383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.898142    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Nov 03 08:08:54.971816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.971776    6430 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:54.974497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.974468    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.975045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:54.972085    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.011776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.011597    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.015280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.015249    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:55.015555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.015474    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.046640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.046100    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 08:08:55.064166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.063169    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.187166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.186110    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.187166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.187039    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.331225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.331167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.378216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.378167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.378965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.378902    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:55.558719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.558619    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.634089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.601121    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.634446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.634402    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Nov 03 08:08:55.648087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.635157    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:08:55.986259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.933300    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:55.986259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.948741    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:08:55.986259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:55.979444    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.069456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.065238    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.069456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.066129    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.078085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.070611    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:08:56.192557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.192514    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.317380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.317292    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:56.317624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.317595    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.340091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.332281    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.493090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.491466    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:08:56.569565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.569474    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:08:56.581261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.581222    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 08:08:56.634218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.634170    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.641255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.637127    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.696461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.689239    6430 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:56.700371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.700332    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.850136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.818630    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:08:56.883189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.854295    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:08:56.883189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.874381    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:08:56.934811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.934772    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.954112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:56.954087    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:08:57.086972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:57.086923    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.388282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:57.388179    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.519794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:57.519746    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:08:57.816919 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:57.816890    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:08:57.890481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:57.889402    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:08:58.378384 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:08:58.378349    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:08:59.885984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:08:59.885947    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:01.080358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:01.080323    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:01.891737 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:01.891696    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:02.660550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:02.660521    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:03.629550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:03.629518    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:04.452518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:04.452488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:05.121292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:05.121258    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:05.947504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:05.947470    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:06.734482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:06.734444    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:07.163430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:07.163346    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:07.277863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:07.277832    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:07.795332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:07.795298    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:07.986990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:07.986954    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"]
Nov 03 08:09:08.080493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:08.080412    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:08.185578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:08.185541    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:08.954037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:08.954003    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:09.172752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:09.172645    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:09.281032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:09.280907    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:10.200120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:10.200056    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:11.239358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:11.239324    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:11.999282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:11.999248    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"]
Nov 03 08:09:11.999795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:11.999760    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj" podUID="fb9c9347-5cd9-4919-9108-f5d40a06bd83" containerName="maas-api" containerID="cri-o://a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401" gracePeriod=30
Nov 03 08:09:12.076313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.076282    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:12.181491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.181460    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"]
Nov 03 08:09:12.181694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.181665    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd" podUID="a60c792b-e425-4583-9cb9-155678a50975" containerName="maas-api" containerID="cri-o://17cc3b3ac6a4ec7d436e9cc86d785025f08ace6b48d434d6aa80c4a8b1d34821" gracePeriod=30
Nov 03 08:09:12.196754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.196492    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:12.310796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.310692    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:12.359343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.359317    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:09:12.411325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.411285    6430 generic.go:356] "Generic (PLEG): container finished" podID="a60c792b-e425-4583-9cb9-155678a50975" containerID="17cc3b3ac6a4ec7d436e9cc86d785025f08ace6b48d434d6aa80c4a8b1d34821" exitCode=0
Nov 03 08:09:12.411490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.411453    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd" event={"ID":"a60c792b-e425-4583-9cb9-155678a50975","Type":"ContainerDied","Data":"17cc3b3ac6a4ec7d436e9cc86d785025f08ace6b48d434d6aa80c4a8b1d34821"}
Nov 03 08:09:12.412856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.412829    6430 generic.go:356] "Generic (PLEG): container finished" podID="fb9c9347-5cd9-4919-9108-f5d40a06bd83" containerID="a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401" exitCode=0
Nov 03 08:09:12.412984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.412914    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj" event={"ID":"fb9c9347-5cd9-4919-9108-f5d40a06bd83","Type":"ContainerDied","Data":"a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401"}
Nov 03 08:09:12.412984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.412945    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj" event={"ID":"fb9c9347-5cd9-4919-9108-f5d40a06bd83","Type":"ContainerDied","Data":"8fa318711e77cf501c5799d95f7394999290b6389fce7cdf62b2145810274c90"}
Nov 03 08:09:12.412984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.412964    6430 scope.go:122] "RemoveContainer" containerID="a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401"
Nov 03 08:09:12.413183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.413100    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"
Nov 03 08:09:12.415626 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.415603    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:12.432858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.432598    6430 scope.go:122] "RemoveContainer" containerID="a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401"
Nov 03 08:09:12.434125 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:12.432969    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401\": container with ID starting with a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401 not found: ID does not exist" containerID="a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401"
Nov 03 08:09:12.434125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.433010    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401"} err="failed to get container status \"a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401\": rpc error: code = NotFound desc = could not find container \"a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401\": container with ID starting with a53fe13cb87ebdb2985131b75c146aa33be428b588b26e1d0886e2f12cc6e401 not found: ID does not exist"
Nov 03 08:09:12.466758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.459988    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9c9347-5cd9-4919-9108-f5d40a06bd83-kube-api-access-8lgqh\" (UniqueName: \"kubernetes.io/projected/fb9c9347-5cd9-4919-9108-f5d40a06bd83-kube-api-access-8lgqh\") pod \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") "
Nov 03 08:09:12.466758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.460284    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls\") pod \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") "
Nov 03 08:09:12.466758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.460385    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9c9347-5cd9-4919-9108-f5d40a06bd83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9c9347-5cd9-4919-9108-f5d40a06bd83-openshift-service-ca\") pod \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") "
Nov 03 08:09:12.466758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.460533    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-metrics-certs\") pod \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\" (UID: \"fb9c9347-5cd9-4919-9108-f5d40a06bd83\") "
Nov 03 08:09:12.466758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.464309    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9c9347-5cd9-4919-9108-f5d40a06bd83-openshift-service-ca" pod "fb9c9347-5cd9-4919-9108-f5d40a06bd83" (UID: "fb9c9347-5cd9-4919-9108-f5d40a06bd83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:12.467354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.467324    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9c9347-5cd9-4919-9108-f5d40a06bd83-kube-api-access-8lgqh" pod "fb9c9347-5cd9-4919-9108-f5d40a06bd83" (UID: "fb9c9347-5cd9-4919-9108-f5d40a06bd83"). InnerVolumeSpecName "kube-api-access-8lgqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:12.469350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.469326    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-metrics-certs" pod "fb9c9347-5cd9-4919-9108-f5d40a06bd83" (UID: "fb9c9347-5cd9-4919-9108-f5d40a06bd83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:12.469913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.469891    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls" pod "fb9c9347-5cd9-4919-9108-f5d40a06bd83" (UID: "fb9c9347-5cd9-4919-9108-f5d40a06bd83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:12.514379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.510964    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:09:12.562867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.562785    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgqh\" (UniqueName: \"kubernetes.io/projected/fb9c9347-5cd9-4919-9108-f5d40a06bd83-kube-api-access-8lgqh\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.562867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.562819    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.562867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.562831    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9c9347-5cd9-4919-9108-f5d40a06bd83-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.562867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.562844    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb9c9347-5cd9-4919-9108-f5d40a06bd83-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.656964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.656932    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:12.681127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.663755    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-maas-api-tls\") pod \"a60c792b-e425-4583-9cb9-155678a50975\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") "
Nov 03 08:09:12.681127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.663855    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-metrics-certs\") pod \"a60c792b-e425-4583-9cb9-155678a50975\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") "
Nov 03 08:09:12.681127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.663929    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60c792b-e425-4583-9cb9-155678a50975-kube-api-access-h2dr8\" (UniqueName: \"kubernetes.io/projected/a60c792b-e425-4583-9cb9-155678a50975-kube-api-access-h2dr8\") pod \"a60c792b-e425-4583-9cb9-155678a50975\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") "
Nov 03 08:09:12.681127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.664024    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60c792b-e425-4583-9cb9-155678a50975-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60c792b-e425-4583-9cb9-155678a50975-openshift-service-ca\") pod \"a60c792b-e425-4583-9cb9-155678a50975\" (UID: \"a60c792b-e425-4583-9cb9-155678a50975\") "
Nov 03 08:09:12.686619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.686274    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60c792b-e425-4583-9cb9-155678a50975-openshift-service-ca" pod "a60c792b-e425-4583-9cb9-155678a50975" (UID: "a60c792b-e425-4583-9cb9-155678a50975"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:12.715117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.703808    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-metrics-certs" pod "a60c792b-e425-4583-9cb9-155678a50975" (UID: "a60c792b-e425-4583-9cb9-155678a50975"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:12.740091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.738275    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60c792b-e425-4583-9cb9-155678a50975-kube-api-access-h2dr8" pod "a60c792b-e425-4583-9cb9-155678a50975" (UID: "a60c792b-e425-4583-9cb9-155678a50975"). InnerVolumeSpecName "kube-api-access-h2dr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:12.771130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.769181    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-maas-api-tls" pod "a60c792b-e425-4583-9cb9-155678a50975" (UID: "a60c792b-e425-4583-9cb9-155678a50975"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:12.783087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.776109    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"]
Nov 03 08:09:12.784271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.784241    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.784399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.784278    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dr8\" (UniqueName: \"kubernetes.io/projected/a60c792b-e425-4583-9cb9-155678a50975-kube-api-access-h2dr8\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.784399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.784296    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60c792b-e425-4583-9cb9-155678a50975-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.784399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.784309    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60c792b-e425-4583-9cb9-155678a50975-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.811994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.811930    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a9dd59a-6c4bd6fcb6-6d6sj"]
Nov 03 08:09:12.863595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.863482    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:12.989775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.989742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:12.990236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:12.990208    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c" podUID="a0084481-266e-4a9c-99a2-e6e9320d9712" containerName="payload-pre-processing" containerID="cri-o://922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281" gracePeriod=25
Nov 03 08:09:13.200017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.198732    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:13.381630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.381597    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:09:13.425007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.424851    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"
Nov 03 08:09:13.425007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.424875    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd" event={"ID":"a60c792b-e425-4583-9cb9-155678a50975","Type":"ContainerDied","Data":"dc6f30801c38c3f27ebfbee3894e39cb33893e6885ca984fd762b5348060216f"}
Nov 03 08:09:13.425464 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.425442    6430 scope.go:122] "RemoveContainer" containerID="17cc3b3ac6a4ec7d436e9cc86d785025f08ace6b48d434d6aa80c4a8b1d34821"
Nov 03 08:09:13.427824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.427803    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0084481-266e-4a9c-99a2-e6e9320d9712-kube-api-access-9qkkg\" (UniqueName: \"kubernetes.io/projected/a0084481-266e-4a9c-99a2-e6e9320d9712-kube-api-access-9qkkg\") pod \"a0084481-266e-4a9c-99a2-e6e9320d9712\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") "
Nov 03 08:09:13.428009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.427991    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0084481-266e-4a9c-99a2-e6e9320d9712-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0084481-266e-4a9c-99a2-e6e9320d9712-plugins-config-volume\") pod \"a0084481-266e-4a9c-99a2-e6e9320d9712\" (UID: \"a0084481-266e-4a9c-99a2-e6e9320d9712\") "
Nov 03 08:09:13.429240 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.428863    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0084481-266e-4a9c-99a2-e6e9320d9712-plugins-config-volume" pod "a0084481-266e-4a9c-99a2-e6e9320d9712" (UID: "a0084481-266e-4a9c-99a2-e6e9320d9712"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:13.430253 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.430226    6430 generic.go:356] "Generic (PLEG): container finished" podID="a0084481-266e-4a9c-99a2-e6e9320d9712" containerID="922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281" exitCode=0
Nov 03 08:09:13.430349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.430307    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"
Nov 03 08:09:13.430349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.430340    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c" event={"ID":"a0084481-266e-4a9c-99a2-e6e9320d9712","Type":"ContainerDied","Data":"922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281"}
Nov 03 08:09:13.430465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.430384    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c" event={"ID":"a0084481-266e-4a9c-99a2-e6e9320d9712","Type":"ContainerDied","Data":"b6b9cf4466eb2935e7e321bb829fd6637ceffe8907e0699b6143196e220714c8"}
Nov 03 08:09:13.432420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.432390    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0084481-266e-4a9c-99a2-e6e9320d9712-kube-api-access-9qkkg" pod "a0084481-266e-4a9c-99a2-e6e9320d9712" (UID: "a0084481-266e-4a9c-99a2-e6e9320d9712"). InnerVolumeSpecName "kube-api-access-9qkkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:13.451645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.451619    6430 scope.go:122] "RemoveContainer" containerID="922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281"
Nov 03 08:09:13.464394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.464359    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"]
Nov 03 08:09:13.470659 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.470635    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c855f06a-74b7555f6c-tmhwd"]
Nov 03 08:09:13.470865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.470851    6430 scope.go:122] "RemoveContainer" containerID="922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281"
Nov 03 08:09:13.471282 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:13.471260    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281\": container with ID starting with 922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281 not found: ID does not exist" containerID="922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281"
Nov 03 08:09:13.471366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.471295    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281"} err="failed to get container status \"922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281\": rpc error: code = NotFound desc = could not find container \"922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281\": container with ID starting with 922f847093a217ac43c8343006df32ebd9ec516ea8d2e4ff289c2fea741db281 not found: ID does not exist"
Nov 03 08:09:13.503868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.503838    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:13.530496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.530455    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0084481-266e-4a9c-99a2-e6e9320d9712-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:13.530496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.530491    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkkg\" (UniqueName: \"kubernetes.io/projected/a0084481-266e-4a9c-99a2-e6e9320d9712-kube-api-access-9qkkg\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:13.624903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.624869    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:13.783637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.783598    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:13.845159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.845124    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"]
Nov 03 08:09:13.848393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:13.848357    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c855f06a-cc5445575-t2x5c"]
Nov 03 08:09:14.626136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:14.623841    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0084481-266e-4a9c-99a2-e6e9320d9712" path="/var/lib/kubelet/pods/a0084481-266e-4a9c-99a2-e6e9320d9712/volumes"
Nov 03 08:09:14.626136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:14.624572    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60c792b-e425-4583-9cb9-155678a50975" path="/var/lib/kubelet/pods/a60c792b-e425-4583-9cb9-155678a50975/volumes"
Nov 03 08:09:14.626136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:14.625203    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9c9347-5cd9-4919-9108-f5d40a06bd83" path="/var/lib/kubelet/pods/fb9c9347-5cd9-4919-9108-f5d40a06bd83/volumes"
Nov 03 08:09:14.694727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:14.694695    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:14.695565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:14.695526    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"]
Nov 03 08:09:14.695796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:14.695769    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w" podUID="50008583-2f67-4f0a-aa77-763cff2726bc" containerName="maas-api" containerID="cri-o://e1adbfb90d116b9caeadb415c4ebf106e1b9e8e56378e6c07673079747124ab6" gracePeriod=30
Nov 03 08:09:14.808435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:14.808401    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.095084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.080354    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.285386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.285326    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.442678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.442540    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.473873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.473565    6430 generic.go:356] "Generic (PLEG): container finished" podID="50008583-2f67-4f0a-aa77-763cff2726bc" containerID="e1adbfb90d116b9caeadb415c4ebf106e1b9e8e56378e6c07673079747124ab6" exitCode=0
Nov 03 08:09:15.473873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.473645    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w" event={"ID":"50008583-2f67-4f0a-aa77-763cff2726bc","Type":"ContainerDied","Data":"e1adbfb90d116b9caeadb415c4ebf106e1b9e8e56378e6c07673079747124ab6"}
Nov 03 08:09:15.473873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.473695    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w" event={"ID":"50008583-2f67-4f0a-aa77-763cff2726bc","Type":"ContainerDied","Data":"22deb319e15ba58ba3ff88810001aeae41951a3d82c5b97d410f55e906bbb847"}
Nov 03 08:09:15.473873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.473718    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22deb319e15ba58ba3ff88810001aeae41951a3d82c5b97d410f55e906bbb847"
Nov 03 08:09:15.521000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.520457    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:09:15.571417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.571368    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.603735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.603698    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"]
Nov 03 08:09:15.604158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.603938    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424" podUID="098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" containerName="istio-proxy" containerID="cri-o://00310e7fe784cd3c2ab322e6671d28ddb0e912f04454101d458ab49fcf1e6704" gracePeriod=30
Nov 03 08:09:15.658832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.658782    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls\") pod \"50008583-2f67-4f0a-aa77-763cff2726bc\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") "
Nov 03 08:09:15.679572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.679545    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-metrics-certs\") pod \"50008583-2f67-4f0a-aa77-763cff2726bc\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") "
Nov 03 08:09:15.679782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.679763    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50008583-2f67-4f0a-aa77-763cff2726bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50008583-2f67-4f0a-aa77-763cff2726bc-openshift-service-ca\") pod \"50008583-2f67-4f0a-aa77-763cff2726bc\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") "
Nov 03 08:09:15.679974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.679954    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50008583-2f67-4f0a-aa77-763cff2726bc-kube-api-access-7f9n5\" (UniqueName: \"kubernetes.io/projected/50008583-2f67-4f0a-aa77-763cff2726bc-kube-api-access-7f9n5\") pod \"50008583-2f67-4f0a-aa77-763cff2726bc\" (UID: \"50008583-2f67-4f0a-aa77-763cff2726bc\") "
Nov 03 08:09:15.681730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.669731    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls" pod "50008583-2f67-4f0a-aa77-763cff2726bc" (UID: "50008583-2f67-4f0a-aa77-763cff2726bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:15.683116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.677483    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"]
Nov 03 08:09:15.684098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684041    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50008583-2f67-4f0a-aa77-763cff2726bc-openshift-service-ca" pod "50008583-2f67-4f0a-aa77-763cff2726bc" (UID: "50008583-2f67-4f0a-aa77-763cff2726bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:15.684253 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684171    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-metrics-certs" pod "50008583-2f67-4f0a-aa77-763cff2726bc" (UID: "50008583-2f67-4f0a-aa77-763cff2726bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:15.684529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684505    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a60c792b-e425-4583-9cb9-155678a50975" containerName="maas-api"
Nov 03 08:09:15.684611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684532    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60c792b-e425-4583-9cb9-155678a50975" containerName="maas-api"
Nov 03 08:09:15.684611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684603    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a0084481-266e-4a9c-99a2-e6e9320d9712" containerName="payload-pre-processing"
Nov 03 08:09:15.684718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684612    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0084481-266e-4a9c-99a2-e6e9320d9712" containerName="payload-pre-processing"
Nov 03 08:09:15.684718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684630    6430 cpu_manager.go:414] "containerMap: removing container" podUID="50008583-2f67-4f0a-aa77-763cff2726bc" containerName="maas-api"
Nov 03 08:09:15.684718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684638    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50008583-2f67-4f0a-aa77-763cff2726bc" containerName="maas-api"
Nov 03 08:09:15.684718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684711    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fb9c9347-5cd9-4919-9108-f5d40a06bd83" containerName="maas-api"
Nov 03 08:09:15.684922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684720    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9c9347-5cd9-4919-9108-f5d40a06bd83" containerName="maas-api"
Nov 03 08:09:15.685017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.684999    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0084481-266e-4a9c-99a2-e6e9320d9712" containerName="payload-pre-processing"
Nov 03 08:09:15.685086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.685029    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="50008583-2f67-4f0a-aa77-763cff2726bc" containerName="maas-api"
Nov 03 08:09:15.685086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.685052    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9c9347-5cd9-4919-9108-f5d40a06bd83" containerName="maas-api"
Nov 03 08:09:15.685086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.685082    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60c792b-e425-4583-9cb9-155678a50975" containerName="maas-api"
Nov 03 08:09:15.689848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.689801    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"]
Nov 03 08:09:15.689960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.689934    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.698618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.698592    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50008583-2f67-4f0a-aa77-763cff2726bc-kube-api-access-7f9n5" pod "50008583-2f67-4f0a-aa77-763cff2726bc" (UID: "50008583-2f67-4f0a-aa77-763cff2726bc"). InnerVolumeSpecName "kube-api-access-7f9n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:15.705567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.705425    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.783376    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c53cba8d-openshift-default-dockercfg-77cq6\""
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.784463    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-certs\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.784505    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-envoy\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.784545    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacd7c89-25a0-4db6-b494-98685ae4724e-istiod-ca-cert\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.784697    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-podinfo\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.784735    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-socket\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.784867    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-token\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.790088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.784980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4vw\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-kube-api-access-qv4vw\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.819084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.818148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-credential-socket\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.819084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.818197    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-data\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.819084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.818459    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:15.819084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.818478    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50008583-2f67-4f0a-aa77-763cff2726bc-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:15.819084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.818495    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50008583-2f67-4f0a-aa77-763cff2726bc-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:15.819084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.818510    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9n5\" (UniqueName: \"kubernetes.io/projected/50008583-2f67-4f0a-aa77-763cff2726bc-kube-api-access-7f9n5\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:15.848084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.837029    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927077    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-certs\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927126    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-envoy\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927186    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacd7c89-25a0-4db6-b494-98685ae4724e-istiod-ca-cert\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927304    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-podinfo\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927338    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-socket\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927688    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-token\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927812    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4vw\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-kube-api-access-qv4vw\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927839    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-credential-socket\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.927862    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-data\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.928375    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-data\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.928596    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-certs\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.929133    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-socket\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.929644    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacd7c89-25a0-4db6-b494-98685ae4724e-istiod-ca-cert\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.931087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.930406    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-credential-socket\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.955768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.939870    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-envoy\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.955768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.947823    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-token\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.955768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.948444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4vw\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-kube-api-access-qv4vw\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:15.957518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.957490    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:15.976459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:15.971753    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-podinfo\") pod \"e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") " pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:16.035088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.029758    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:16.482195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.482158    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"
Nov 03 08:09:16.512863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.512834    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"]
Nov 03 08:09:16.516277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.516253    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b397f45-5875844bcd-ftq2w"]
Nov 03 08:09:16.619256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.619221    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50008583-2f67-4f0a-aa77-763cff2726bc" path="/var/lib/kubelet/pods/50008583-2f67-4f0a-aa77-763cff2726bc/volumes"
Nov 03 08:09:16.623562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.623529    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"]
Nov 03 08:09:16.625679 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:09:16.625656    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd7c89_25a0_4db6_b494_98685ae4724e.slice/crio-d5cf012871ee887984301e9374db0e02ba19dc90705b4a44127094b5edecfe41 WatchSource:0}: Error finding container d5cf012871ee887984301e9374db0e02ba19dc90705b4a44127094b5edecfe41: Status 404 returned error can't find the container with id d5cf012871ee887984301e9374db0e02ba19dc90705b4a44127094b5edecfe41
Nov 03 08:09:16.628208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.628176    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:09:16.628307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.628243    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:09:16.628307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.628274    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:09:16.856108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:16.854367    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:17.026183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:17.026140    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:17.162260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:17.162210    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:17.346672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:17.346594    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:17.570121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:17.568279    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:17.585095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:17.576520    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr" event={"ID":"cacd7c89-25a0-4db6-b494-98685ae4724e","Type":"ContainerStarted","Data":"69fa83bbe0a057811fd294304b74914f0248dff581c3690f4e543cdd5913720f"}
Nov 03 08:09:17.585095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:17.576571    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr" event={"ID":"cacd7c89-25a0-4db6-b494-98685ae4724e","Type":"ContainerStarted","Data":"d5cf012871ee887984301e9374db0e02ba19dc90705b4a44127094b5edecfe41"}
Nov 03 08:09:17.630160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:17.622991    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr" podStartSLOduration=2.62297161 podStartE2EDuration="2.62297161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:17.608389136 +0000 UTC m=+907.111224653" watchObservedRunningTime="2026-11-03 08:09:17.62297161 +0000 UTC m=+907.125807119"
Nov 03 08:09:18.030640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.030608    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:18.035870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.035846    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:18.264663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.264635    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:18.309917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.309845    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"]
Nov 03 08:09:18.315055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.315032    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:18.321106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.320279    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9b241313\""
Nov 03 08:09:18.321106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.320398    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9b241313-dockercfg-cbrxr\""
Nov 03 08:09:18.321404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.321380    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"]
Nov 03 08:09:18.360999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.360961    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pcz\" (UniqueName: \"kubernetes.io/projected/211eaa2f-16f4-4e48-8060-7aa70039c7bd-kube-api-access-s2pcz\") pod \"payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:18.361187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.361145    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eaa2f-16f4-4e48-8060-7aa70039c7bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:18.371943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.371913    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"]
Nov 03 08:09:18.376894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.376867    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:18.382284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.382262    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"]
Nov 03 08:09:18.389361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.389334    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:18.477086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.466994    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") " pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:18.477086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.467094    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pcz\" (UniqueName: \"kubernetes.io/projected/211eaa2f-16f4-4e48-8060-7aa70039c7bd-kube-api-access-s2pcz\") pod \"payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:18.477086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.467188    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eaa2f-16f4-4e48-8060-7aa70039c7bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:18.477086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.467253    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z875t\" (UniqueName: \"kubernetes.io/projected/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-kube-api-access-z875t\") pod \"payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") " pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:18.477086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.468825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eaa2f-16f4-4e48-8060-7aa70039c7bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:18.536471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.533505    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:18.632147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.627637    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") " pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:18.632147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.627823    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z875t\" (UniqueName: \"kubernetes.io/projected/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-kube-api-access-z875t\") pod \"payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") " pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:18.632147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.629265    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") " pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:18.699445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.699347    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pcz\" (UniqueName: \"kubernetes.io/projected/211eaa2f-16f4-4e48-8060-7aa70039c7bd-kube-api-access-s2pcz\") pod \"payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:18.726621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.726589    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z875t\" (UniqueName: \"kubernetes.io/projected/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-kube-api-access-z875t\") pod \"payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") " pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:18.821523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.821460    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:18.824462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.824437    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:09:18.953475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:18.953439    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:19.023847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.002330    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:19.420034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.419643    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"]
Nov 03 08:09:19.429106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.429077    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"]
Nov 03 08:09:19.507125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.507094    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:19.631225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.631138    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:19.793888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.792732    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb" event={"ID":"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f","Type":"ContainerStarted","Data":"cdbad2130edadd3d9fc377d5a399d2dbda9cc8459e19f484b3f0df1b9c309057"}
Nov 03 08:09:19.793888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.792796    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb" event={"ID":"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f","Type":"ContainerStarted","Data":"9c37d0230420288cc4a0c77efb6ebc9dac6e9e2fcdc0a8eccc31094710d9f104"}
Nov 03 08:09:19.793888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.793814    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:19.838434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.838374    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb" podStartSLOduration=1.838350738 podStartE2EDuration="1.838350738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:19.815430975 +0000 UTC m=+909.318266516" watchObservedRunningTime="2026-11-03 08:09:19.838350738 +0000 UTC m=+909.341186251"
Nov 03 08:09:19.871133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.866595    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk" event={"ID":"211eaa2f-16f4-4e48-8060-7aa70039c7bd","Type":"ContainerStarted","Data":"891ed2df178857d88ea98290a7eff3db6f589f528ff3798c309833880c1da44c"}
Nov 03 08:09:19.871133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.866664    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk" event={"ID":"211eaa2f-16f4-4e48-8060-7aa70039c7bd","Type":"ContainerStarted","Data":"5b3ac5c1f0d6e13f5d5d8cb1417d49b15a151f2c284914e56eba920275dde6ab"}
Nov 03 08:09:19.871133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.866719    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:19.893096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:19.888252    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk" podStartSLOduration=1.8882329310000001 podStartE2EDuration="1.888232931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:19.885937406 +0000 UTC m=+909.388772920" watchObservedRunningTime="2026-11-03 08:09:19.888232931 +0000 UTC m=+909.391068446"
Nov 03 08:09:20.701805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.701746    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:20.823096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.820092    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"]
Nov 03 08:09:20.825108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.824911    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:20.831434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.831317    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:20.833933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.833864    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c53cba8d\""
Nov 03 08:09:20.833933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.833870    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c53cba8d-dockercfg-48l7r\""
Nov 03 08:09:20.840251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.838641    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"]
Nov 03 08:09:20.886958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.886872    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"]
Nov 03 08:09:20.892476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.892448    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:20.899921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.899523    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"]
Nov 03 08:09:20.914086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.902292    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b005dc-4614-45bd-b172-70f28c6e8638-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:20.914086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.906303    6430 generic.go:356] "Generic (PLEG): container finished" podID="098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" containerID="00310e7fe784cd3c2ab322e6671d28ddb0e912f04454101d458ab49fcf1e6704" exitCode=0
Nov 03 08:09:20.915032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.914411    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424" event={"ID":"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e","Type":"ContainerDied","Data":"00310e7fe784cd3c2ab322e6671d28ddb0e912f04454101d458ab49fcf1e6704"}
Nov 03 08:09:20.915032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:20.914560    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7fh\" (UniqueName: \"kubernetes.io/projected/a8b005dc-4614-45bd-b172-70f28c6e8638-kube-api-access-xl7fh\") pod \"payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:21.017471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.017434    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjz6v\" (UniqueName: \"kubernetes.io/projected/31cb25d8-4419-4758-9a1c-36415878e948-kube-api-access-zjz6v\") pod \"payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") " pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.017851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.017826    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b005dc-4614-45bd-b172-70f28c6e8638-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:21.017976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.017959    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb25d8-4419-4758-9a1c-36415878e948-plugins-config-volume\") pod \"payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") " pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.018202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.018185    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7fh\" (UniqueName: \"kubernetes.io/projected/a8b005dc-4614-45bd-b172-70f28c6e8638-kube-api-access-xl7fh\") pod \"payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:21.026087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.019715    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:21.026087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.021453    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b005dc-4614-45bd-b172-70f28c6e8638-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:21.152097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.150772    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb25d8-4419-4758-9a1c-36415878e948-plugins-config-volume\") pod \"payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") " pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.152097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.151361    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjz6v\" (UniqueName: \"kubernetes.io/projected/31cb25d8-4419-4758-9a1c-36415878e948-kube-api-access-zjz6v\") pod \"payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") " pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.153641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.153603    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb25d8-4419-4758-9a1c-36415878e948-plugins-config-volume\") pod \"payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") " pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.266855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.263279    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:09:21.293978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.293775    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjz6v\" (UniqueName: \"kubernetes.io/projected/31cb25d8-4419-4758-9a1c-36415878e948-kube-api-access-zjz6v\") pod \"payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") " pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.293978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.293909    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:21.356087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.347864    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7fh\" (UniqueName: \"kubernetes.io/projected/a8b005dc-4614-45bd-b172-70f28c6e8638-kube-api-access-xl7fh\") pod \"payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:21.384558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384508    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-token\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-token\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.384727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384598    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-socket\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.384727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384637    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-podinfo\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.384727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384675    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-certs\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.384925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384734    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-data\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.384925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384776    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istiod-ca-cert\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.385032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384941    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-envoy\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.385032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.384998    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-kube-api-access-hsgfz\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-kube-api-access-hsgfz\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.385160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.385034    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-credential-socket\") pod \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\" (UID: \"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e\") "
Nov 03 08:09:21.393543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.393439    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istiod-ca-cert" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:21.396246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.394436    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-socket" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:09:21.396406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.394462    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-data" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:09:21.396486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.395414    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-token" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:21.397345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.397301    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-certs" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:09:21.397924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.397898    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-credential-socket" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:09:21.408169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.407236    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-kube-api-access-hsgfz" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "kube-api-access-hsgfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:21.413161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.411121    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:21.413161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.412836    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-envoy" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:09:21.415736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.415705    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-podinfo" pod "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" (UID: "098a2d98-a3f0-42f2-b9ca-f9b60eceed0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:09:21.453230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.453191    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:21.487452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487425    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487456    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487467    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487476    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487485    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487494    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487502    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487511    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsgfz\" (UniqueName: \"kubernetes.io/projected/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-kube-api-access-hsgfz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.487579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.487527    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:21.542001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.541977    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.737525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.737498    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"]
Nov 03 08:09:21.739545 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:09:21.739503    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b005dc_4614_45bd_b172_70f28c6e8638.slice/crio-e09891d803cd92cd9ed30bf9b4baa08c5fa0999c7c0d60037955a0b3e88db415 WatchSource:0}: Error finding container e09891d803cd92cd9ed30bf9b4baa08c5fa0999c7c0d60037955a0b3e88db415: Status 404 returned error can't find the container with id e09891d803cd92cd9ed30bf9b4baa08c5fa0999c7c0d60037955a0b3e88db415
Nov 03 08:09:21.832132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.832104    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"]
Nov 03 08:09:21.834198 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:09:21.834177    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cb25d8_4419_4758_9a1c_36415878e948.slice/crio-7aac0bb5ae636f574c27933b9894bb0295c9e5fa590434d2ae4a809859652093 WatchSource:0}: Error finding container 7aac0bb5ae636f574c27933b9894bb0295c9e5fa590434d2ae4a809859652093: Status 404 returned error can't find the container with id 7aac0bb5ae636f574c27933b9894bb0295c9e5fa590434d2ae4a809859652093
Nov 03 08:09:21.952322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.952289    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424" event={"ID":"098a2d98-a3f0-42f2-b9ca-f9b60eceed0e","Type":"ContainerDied","Data":"ee44b1ff6598c39a367f72feeeef1cc63a6d4ee9604b327adaf394246099190e"}
Nov 03 08:09:21.952322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.952307    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"
Nov 03 08:09:21.952556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.952361    6430 scope.go:122] "RemoveContainer" containerID="00310e7fe784cd3c2ab322e6671d28ddb0e912f04454101d458ab49fcf1e6704"
Nov 03 08:09:21.956520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.956486    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr" event={"ID":"31cb25d8-4419-4758-9a1c-36415878e948","Type":"ContainerStarted","Data":"07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a"}
Nov 03 08:09:21.956687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.956548    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr" event={"ID":"31cb25d8-4419-4758-9a1c-36415878e948","Type":"ContainerStarted","Data":"7aac0bb5ae636f574c27933b9894bb0295c9e5fa590434d2ae4a809859652093"}
Nov 03 08:09:21.957394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.957371    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:21.959520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.959286    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq" event={"ID":"a8b005dc-4614-45bd-b172-70f28c6e8638","Type":"ContainerStarted","Data":"190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051"}
Nov 03 08:09:21.959520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.959344    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq" event={"ID":"a8b005dc-4614-45bd-b172-70f28c6e8638","Type":"ContainerStarted","Data":"e09891d803cd92cd9ed30bf9b4baa08c5fa0999c7c0d60037955a0b3e88db415"}
Nov 03 08:09:21.959520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:21.959432    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:22.008488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.008394    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq" podStartSLOduration=2.008374274 podStartE2EDuration="2.008374274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:22.002193911 +0000 UTC m=+911.505029425" watchObservedRunningTime="2026-11-03 08:09:22.008374274 +0000 UTC m=+911.511209788"
Nov 03 08:09:22.031417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.031366    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr" podStartSLOduration=2.031353467 podStartE2EDuration="2.031353467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:22.030209813 +0000 UTC m=+911.533045328" watchObservedRunningTime="2026-11-03 08:09:22.031353467 +0000 UTC m=+911.534188981"
Nov 03 08:09:22.051863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.051834    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"]
Nov 03 08:09:22.058430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.058393    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0b397f45-gw-openshift-default-55cbff7649-hp424"]
Nov 03 08:09:22.070782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.070742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:22.575880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.575853    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:22.618141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.618110    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" path="/var/lib/kubelet/pods/098a2d98-a3f0-42f2-b9ca-f9b60eceed0e/volumes"
Nov 03 08:09:22.622266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.622240    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"]
Nov 03 08:09:22.622846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.622834    6430 cpu_manager.go:414] "containerMap: removing container" podUID="098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" containerName="istio-proxy"
Nov 03 08:09:22.622896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.622848    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" containerName="istio-proxy"
Nov 03 08:09:22.622961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.622952    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="098a2d98-a3f0-42f2-b9ca-f9b60eceed0e" containerName="istio-proxy"
Nov 03 08:09:22.627769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.627753    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.633239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.633220    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-a60004\""
Nov 03 08:09:22.634964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.634942    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"]
Nov 03 08:09:22.660450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.660427    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-a60004-dockercfg-w4fl2\""
Nov 03 08:09:22.676409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.676387    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"]
Nov 03 08:09:22.680496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.680480    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:22.689461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.689439    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"]
Nov 03 08:09:22.701456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.701435    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:22.710258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.710217    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47341871-69c7-411a-ab2e-8463095626e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.710605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.710579    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vglkf\" (UniqueName: \"kubernetes.io/projected/47341871-69c7-411a-ab2e-8463095626e0-kube-api-access-vglkf\") pod \"payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.858148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.852199    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47341871-69c7-411a-ab2e-8463095626e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.858148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.852287    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvjc\" (UniqueName: \"kubernetes.io/projected/47e8aa56-70f1-4832-b52d-89a3ee495556-kube-api-access-4rvjc\") pod \"payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:22.858148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.852340    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e8aa56-70f1-4832-b52d-89a3ee495556-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:22.858148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.852438    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vglkf\" (UniqueName: \"kubernetes.io/projected/47341871-69c7-411a-ab2e-8463095626e0-kube-api-access-vglkf\") pod \"payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.858148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.853678    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47341871-69c7-411a-ab2e-8463095626e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.913698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.883808    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vglkf\" (UniqueName: \"kubernetes.io/projected/47341871-69c7-411a-ab2e-8463095626e0-kube-api-access-vglkf\") pod \"payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.978084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.952674    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:22.978084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.953688    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvjc\" (UniqueName: \"kubernetes.io/projected/47e8aa56-70f1-4832-b52d-89a3ee495556-kube-api-access-4rvjc\") pod \"payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:22.978084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.954146    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e8aa56-70f1-4832-b52d-89a3ee495556-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:22.978084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:22.955261    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e8aa56-70f1-4832-b52d-89a3ee495556-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:23.040668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:23.039830    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvjc\" (UniqueName: \"kubernetes.io/projected/47e8aa56-70f1-4832-b52d-89a3ee495556-kube-api-access-4rvjc\") pod \"payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:23.255757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:23.255725    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:23.305430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:23.305394    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:23.411129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:23.411094    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"]
Nov 03 08:09:23.415893 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:09:23.415861    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47341871_69c7_411a_ab2e_8463095626e0.slice/crio-c0266df3a1f7ea63ba677bc76799d894d4da187065c4967e801d6bfdb06eefae WatchSource:0}: Error finding container c0266df3a1f7ea63ba677bc76799d894d4da187065c4967e801d6bfdb06eefae: Status 404 returned error can't find the container with id c0266df3a1f7ea63ba677bc76799d894d4da187065c4967e801d6bfdb06eefae
Nov 03 08:09:23.676444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:23.676409    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"]
Nov 03 08:09:23.678210 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:09:23.678184    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e8aa56_70f1_4832_b52d_89a3ee495556.slice/crio-12f247f58a7070a13e6e358fb62ef460c3097343223d5043e179f8ecbb69c9e5 WatchSource:0}: Error finding container 12f247f58a7070a13e6e358fb62ef460c3097343223d5043e179f8ecbb69c9e5: Status 404 returned error can't find the container with id 12f247f58a7070a13e6e358fb62ef460c3097343223d5043e179f8ecbb69c9e5
Nov 03 08:09:24.045702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.045668    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:24.063934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.063904    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj" event={"ID":"47e8aa56-70f1-4832-b52d-89a3ee495556","Type":"ContainerStarted","Data":"ea4c27824be282b6315325ddba52b1928da19f6810e48e2fe761f4f3c9f42a1b"}
Nov 03 08:09:24.064192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.063960    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj" event={"ID":"47e8aa56-70f1-4832-b52d-89a3ee495556","Type":"ContainerStarted","Data":"12f247f58a7070a13e6e358fb62ef460c3097343223d5043e179f8ecbb69c9e5"}
Nov 03 08:09:24.064192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.063993    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:24.065998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.065967    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2" event={"ID":"47341871-69c7-411a-ab2e-8463095626e0","Type":"ContainerStarted","Data":"73a25e38f7d5476fe5ccf4c0f7f915fed39bd5d932e3eede5a856a558abf5cd7"}
Nov 03 08:09:24.066155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.066020    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2" event={"ID":"47341871-69c7-411a-ab2e-8463095626e0","Type":"ContainerStarted","Data":"c0266df3a1f7ea63ba677bc76799d894d4da187065c4967e801d6bfdb06eefae"}
Nov 03 08:09:24.066211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.066185    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:24.141098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.138603    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj" podStartSLOduration=2.1385825880000002 podStartE2EDuration="2.138582588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:24.129997725 +0000 UTC m=+913.632833239" watchObservedRunningTime="2026-11-03 08:09:24.138582588 +0000 UTC m=+913.641418106"
Nov 03 08:09:24.188330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.188230    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2" podStartSLOduration=2.18820962 podStartE2EDuration="2.18820962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:24.152580455 +0000 UTC m=+913.655415970" watchObservedRunningTime="2026-11-03 08:09:24.18820962 +0000 UTC m=+913.691045135"
Nov 03 08:09:24.736997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:24.736953    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:25.359935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.359902    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"]
Nov 03 08:09:25.364544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.364522    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.370718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.370682    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9b241313-serving-cert\""
Nov 03 08:09:25.377456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.376081    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"]
Nov 03 08:09:25.433826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.433793    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:25.438100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.438041    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a617889-a041-4f62-aa92-efca6335917a-openshift-service-ca\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.438299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.438283    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-metrics-certs\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.438443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.438425    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-maas-api-tls\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.438485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.438473    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbcwq\" (UniqueName: \"kubernetes.io/projected/9a617889-a041-4f62-aa92-efca6335917a-kube-api-access-vbcwq\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.557280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.547547    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:25.557280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.548867    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-metrics-certs\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.557280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.549027    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-maas-api-tls\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.557280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.549114    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbcwq\" (UniqueName: \"kubernetes.io/projected/9a617889-a041-4f62-aa92-efca6335917a-kube-api-access-vbcwq\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.557280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.549184    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a617889-a041-4f62-aa92-efca6335917a-openshift-service-ca\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.557280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.550304    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a617889-a041-4f62-aa92-efca6335917a-openshift-service-ca\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.619136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.608676    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-maas-api-tls\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.636089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.631149    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbcwq\" (UniqueName: \"kubernetes.io/projected/9a617889-a041-4f62-aa92-efca6335917a-kube-api-access-vbcwq\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.643577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.636345    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-metrics-certs\") pod \"maas-api-e2e-ait-9b241313-75649cc5c6-6cq62\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:25.685768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:25.684413    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:26.121386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:26.121360    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"]
Nov 03 08:09:26.263091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:26.263035    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:27.138149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:27.137999    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62" event={"ID":"9a617889-a041-4f62-aa92-efca6335917a","Type":"ContainerStarted","Data":"a7d102a4fbb4f200f00e655003862dd66407b2dcfe9ca7e5ccfceb6f3c772d51"}
Nov 03 08:09:27.138149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:27.138086    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62" event={"ID":"9a617889-a041-4f62-aa92-efca6335917a","Type":"ContainerStarted","Data":"64a0255be019c7fc6b20a0d36ffdd22d02b53e818b917f11b3d7dcd39b9e3f63"}
Nov 03 08:09:27.138149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:27.138118    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:27.155988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:27.155932    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62" podStartSLOduration=1.476159913 podStartE2EDuration="2.155915387s" totalImagesPullingTime="679.755474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:27.155561268 +0000 UTC m=+916.658396783" watchObservedRunningTime="2026-11-03 08:09:27.155915387 +0000 UTC m=+916.658750903"
Nov 03 08:09:27.327236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:27.327201    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:27.429618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:27.429543    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:27.542151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:27.542113    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:28.472629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:28.472571    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:28.661590 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:28.661544    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:09:29.537643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:29.537605    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:30.764086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:30.755267    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:30.904086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:30.880666    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:31.125762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.125671    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-66kgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:09:31.126075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.126017    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-66kgk" podUID="7a5a8bac-1e6a-4fb8-9852-abc001854117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:09:31.208100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.197249    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"]
Nov 03 08:09:31.253337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.253299    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"]
Nov 03 08:09:31.253783 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.253710    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.285966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.281890    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-a60004-serving-cert\""
Nov 03 08:09:31.397179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.397045    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.397586 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.397553    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.397918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.397895    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec8f62e-72ce-485b-868e-c142e9d14291-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.398204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.398184    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbx2\" (UniqueName: \"kubernetes.io/projected/9ec8f62e-72ce-485b-868e-c142e9d14291-kube-api-access-wvbx2\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.545085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.524258    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec8f62e-72ce-485b-868e-c142e9d14291-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.545085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.524341    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbx2\" (UniqueName: \"kubernetes.io/projected/9ec8f62e-72ce-485b-868e-c142e9d14291-kube-api-access-wvbx2\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.545085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.524426    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.545085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.524572    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.545085 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:31.524811    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-serving-cert: secret "maas-api-e2e-shared-a-w0-a60004-serving-cert" not found
Nov 03 08:09:31.545085 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:31.524880    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls podName:9ec8f62e-72ce-485b-868e-c142e9d14291 nodeName:}" failed. No retries permitted until 2026-11-03 08:09:32.024858915 +0000 UTC m=+921.527694407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls") pod "maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd" (UID: "9ec8f62e-72ce-485b-868e-c142e9d14291") : secret "maas-api-e2e-shared-a-w0-a60004-serving-cert" not found
Nov 03 08:09:31.545085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.525995    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec8f62e-72ce-485b-868e-c142e9d14291-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.612102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.610171    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbx2\" (UniqueName: \"kubernetes.io/projected/9ec8f62e-72ce-485b-868e-c142e9d14291-kube-api-access-wvbx2\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.659949 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.659856    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:09:31.665165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.659158    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:31.867087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.862010    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:09:31.928146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.927656    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:31.995287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:31.994121    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:32.000869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:32.000662    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:32.055942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:32.052931    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:32.089915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:32.080598    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:32.236160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:32.230380    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:32.288083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:32.276285    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:32.716969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:32.716949    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"]
Nov 03 08:09:32.719199 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:09:32.719175    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec8f62e_72ce_485b_868e_c142e9d14291.slice/crio-25c480d0643d5656ca816fd14d63701d88ced1d2d419d2e5da68a7fd0c92a94a WatchSource:0}: Error finding container 25c480d0643d5656ca816fd14d63701d88ced1d2d419d2e5da68a7fd0c92a94a: Status 404 returned error can't find the container with id 25c480d0643d5656ca816fd14d63701d88ced1d2d419d2e5da68a7fd0c92a94a
Nov 03 08:09:32.934838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:32.934802    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:33.073778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:33.073748    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:33.193154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:33.193131    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:33.458072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:33.457965    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd" event={"ID":"9ec8f62e-72ce-485b-868e-c142e9d14291","Type":"ContainerStarted","Data":"0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4"}
Nov 03 08:09:33.458072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:33.458049    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:33.458309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:33.458084    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd" event={"ID":"9ec8f62e-72ce-485b-868e-c142e9d14291","Type":"ContainerStarted","Data":"25c480d0643d5656ca816fd14d63701d88ced1d2d419d2e5da68a7fd0c92a94a"}
Nov 03 08:09:33.474535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:33.474489    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd" podStartSLOduration=3.061596957 podStartE2EDuration="3.474475391s" totalImagesPullingTime="412.878434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:33.472617381 +0000 UTC m=+922.975452889" watchObservedRunningTime="2026-11-03 08:09:33.474475391 +0000 UTC m=+922.977310904"
Nov 03 08:09:33.739813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:33.739720    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:34.069515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:34.069436    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:34.436748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:34.436714    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:35.076563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:35.076540    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:09:35.076992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:35.076600    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:09:35.154364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:35.154333    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:35.847669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:35.847625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:36.754082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:36.750698    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:37.636679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:37.636644    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:38.536832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:38.536765    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:39.505958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:39.505931    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:09:39.528552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:39.528525    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:40.309916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:40.309881    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:41.341047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:41.341019    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:42.185365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:42.185315    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:42.683713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:42.683678    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:42.788732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:42.788685    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:42.989355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:42.989245    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:44.010941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:44.010902    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:44.749241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:44.749206    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:45.441259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:45.441231    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:46.435605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:46.435577    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:46.572146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:46.566696    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:46.879481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:46.879397    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:47.377047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:47.377010    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"]
Nov 03 08:09:47.522093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:47.514268    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:47.796543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:47.796509    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"]
Nov 03 08:09:47.820171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:47.802780    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:47.822134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:47.806735    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:47.939100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:47.928274    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-fe584a-openshift-default-dockercfg-74sh7\""
Nov 03 08:09:48.016529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.016481    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-token\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.016847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.016819    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgg5s\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-kube-api-access-zgg5s\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.017082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.017042    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-envoy\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.017437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.017415    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-socket\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.023090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.017605    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-podinfo\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.023371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.023346    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istiod-ca-cert\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.023588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.023570    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-certs\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.023837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.023820    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-credential-socket\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.023957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.023942    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-data\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.053809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.053706    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125475    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-credential-socket\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125527    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-data\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125591    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-token\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125660    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgg5s\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-kube-api-access-zgg5s\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125773    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-envoy\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125813    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-socket\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-podinfo\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.125944    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istiod-ca-cert\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.126031    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-certs\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.126548    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-certs\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.126795    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-credential-socket\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.127001    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-data\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.130171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.130144    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istiod-ca-cert\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.142173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.141556    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-socket\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.171710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.171643    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-podinfo\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.174699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.174633    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-token\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.175117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.175093    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-envoy\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.175858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.175836    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgg5s\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-kube-api-access-zgg5s\") pod \"e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") " pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.186596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.186337    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:48.454787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.454639    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:48.658140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.657424    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"]
Nov 03 08:09:48.689762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.689158    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:09:48.689762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.689257    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:09:48.689762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.689313    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:09:48.841937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.841907    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" event={"ID":"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda","Type":"ContainerStarted","Data":"14bfff959d007fa00cca4446875837aab2b419670c3e949a9b20f5983f20f76f"}
Nov 03 08:09:48.842432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.841957    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" event={"ID":"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda","Type":"ContainerStarted","Data":"931d3083aab0b8c9cc56c4acd038d47ae7a5f17068f403ac7dee83d361b1fac4"}
Nov 03 08:09:48.864157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:48.864093    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podStartSLOduration=1.864055536 podStartE2EDuration="1.864055536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:48.85803535 +0000 UTC m=+938.360870866" watchObservedRunningTime="2026-11-03 08:09:48.864055536 +0000 UTC m=+938.366891051"
Nov 03 08:09:49.202288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:49.202258    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:49.202664 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:49.202637    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 08:09:49.202859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:49.202831    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 08:09:49.242258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:49.242226    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"]
Nov 03 08:09:49.256742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:49.256708    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"]
Nov 03 08:09:49.344625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:49.344522    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:49.591472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:49.591371    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:50.193115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:50.193081    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:50.483041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:50.482957    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:50.890877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:50.890779    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:50.892047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:50.892021    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:09:51.175758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:51.175672    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:51.303585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:51.303549    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:51.548699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:51.548665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:52.424365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:52.424331    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:52.543853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:52.543757    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:53.226150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:53.226113    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"]
Nov 03 08:09:53.229282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:53.229256    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"]
Nov 03 08:09:53.320907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:53.320858    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:53.436079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:53.436038    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:54.248170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.248126    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr" podUID="31cb25d8-4419-4758-9a1c-36415878e948" containerName="payload-processing" containerID="cri-o://07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a" gracePeriod=25
Nov 03 08:09:54.249738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.249175    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:54.257360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.257318    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq" podUID="a8b005dc-4614-45bd-b172-70f28c6e8638" containerName="payload-pre-processing" containerID="cri-o://190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051" gracePeriod=25
Nov 03 08:09:54.628849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.628826    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:54.666611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.666581    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:54.794214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.794124    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cb25d8-4419-4758-9a1c-36415878e948-kube-api-access-zjz6v\" (UniqueName: \"kubernetes.io/projected/31cb25d8-4419-4758-9a1c-36415878e948-kube-api-access-zjz6v\") pod \"31cb25d8-4419-4758-9a1c-36415878e948\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") "
Nov 03 08:09:54.794361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.794320    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31cb25d8-4419-4758-9a1c-36415878e948-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb25d8-4419-4758-9a1c-36415878e948-plugins-config-volume\") pod \"31cb25d8-4419-4758-9a1c-36415878e948\" (UID: \"31cb25d8-4419-4758-9a1c-36415878e948\") "
Nov 03 08:09:54.794418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.794359    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b005dc-4614-45bd-b172-70f28c6e8638-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b005dc-4614-45bd-b172-70f28c6e8638-plugins-config-volume\") pod \"a8b005dc-4614-45bd-b172-70f28c6e8638\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") "
Nov 03 08:09:54.794482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.794454    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b005dc-4614-45bd-b172-70f28c6e8638-kube-api-access-xl7fh\" (UniqueName: \"kubernetes.io/projected/a8b005dc-4614-45bd-b172-70f28c6e8638-kube-api-access-xl7fh\") pod \"a8b005dc-4614-45bd-b172-70f28c6e8638\" (UID: \"a8b005dc-4614-45bd-b172-70f28c6e8638\") "
Nov 03 08:09:54.794761 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.794732    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cb25d8-4419-4758-9a1c-36415878e948-plugins-config-volume" pod "31cb25d8-4419-4758-9a1c-36415878e948" (UID: "31cb25d8-4419-4758-9a1c-36415878e948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:54.794865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.794788    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b005dc-4614-45bd-b172-70f28c6e8638-plugins-config-volume" pod "a8b005dc-4614-45bd-b172-70f28c6e8638" (UID: "a8b005dc-4614-45bd-b172-70f28c6e8638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:54.795134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.795115    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31cb25d8-4419-4758-9a1c-36415878e948-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:54.795134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.795135    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b005dc-4614-45bd-b172-70f28c6e8638-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:54.796341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.796321    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cb25d8-4419-4758-9a1c-36415878e948-kube-api-access-zjz6v" pod "31cb25d8-4419-4758-9a1c-36415878e948" (UID: "31cb25d8-4419-4758-9a1c-36415878e948"). InnerVolumeSpecName "kube-api-access-zjz6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:54.796690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.796669    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b005dc-4614-45bd-b172-70f28c6e8638-kube-api-access-xl7fh" pod "a8b005dc-4614-45bd-b172-70f28c6e8638" (UID: "a8b005dc-4614-45bd-b172-70f28c6e8638"). InnerVolumeSpecName "kube-api-access-xl7fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:54.897021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.896988    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7fh\" (UniqueName: \"kubernetes.io/projected/a8b005dc-4614-45bd-b172-70f28c6e8638-kube-api-access-xl7fh\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:54.897021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.897017    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjz6v\" (UniqueName: \"kubernetes.io/projected/31cb25d8-4419-4758-9a1c-36415878e948-kube-api-access-zjz6v\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:54.930709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.930677    6430 generic.go:356] "Generic (PLEG): container finished" podID="a8b005dc-4614-45bd-b172-70f28c6e8638" containerID="190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051" exitCode=0
Nov 03 08:09:54.930866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.930723    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"
Nov 03 08:09:54.930866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.930759    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq" event={"ID":"a8b005dc-4614-45bd-b172-70f28c6e8638","Type":"ContainerDied","Data":"190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051"}
Nov 03 08:09:54.930866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.930817    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq" event={"ID":"a8b005dc-4614-45bd-b172-70f28c6e8638","Type":"ContainerDied","Data":"e09891d803cd92cd9ed30bf9b4baa08c5fa0999c7c0d60037955a0b3e88db415"}
Nov 03 08:09:54.930866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.930845    6430 scope.go:122] "RemoveContainer" containerID="190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051"
Nov 03 08:09:54.932546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.932525    6430 generic.go:356] "Generic (PLEG): container finished" podID="31cb25d8-4419-4758-9a1c-36415878e948" containerID="07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a" exitCode=0
Nov 03 08:09:54.932716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.932566    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr" event={"ID":"31cb25d8-4419-4758-9a1c-36415878e948","Type":"ContainerDied","Data":"07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a"}
Nov 03 08:09:54.932800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.932746    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"
Nov 03 08:09:54.932859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.932751    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr" event={"ID":"31cb25d8-4419-4758-9a1c-36415878e948","Type":"ContainerDied","Data":"7aac0bb5ae636f574c27933b9894bb0295c9e5fa590434d2ae4a809859652093"}
Nov 03 08:09:54.943874 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.943854    6430 scope.go:122] "RemoveContainer" containerID="190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051"
Nov 03 08:09:54.944178 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:54.944158    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051\": container with ID starting with 190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051 not found: ID does not exist" containerID="190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051"
Nov 03 08:09:54.944242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.944186    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051"} err="failed to get container status \"190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051\": rpc error: code = NotFound desc = could not find container \"190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051\": container with ID starting with 190f0a1c450e0309e97dbd219d437f5e2a7b27f8fa90dcc8fdd4978342a1b051 not found: ID does not exist"
Nov 03 08:09:54.944242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.944202    6430 scope.go:122] "RemoveContainer" containerID="07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a"
Nov 03 08:09:54.954664 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.954646    6430 scope.go:122] "RemoveContainer" containerID="07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a"
Nov 03 08:09:54.954993 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:54.954969    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a\": container with ID starting with 07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a not found: ID does not exist" containerID="07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a"
Nov 03 08:09:54.955123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.955003    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a"} err="failed to get container status \"07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a\": rpc error: code = NotFound desc = could not find container \"07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a\": container with ID starting with 07f93e6284b181acd107c6a0f6975f0b0c6dd4497ec41f2689651ee0d405051a not found: ID does not exist"
Nov 03 08:09:54.955338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.955319    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"]
Nov 03 08:09:54.960260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.960241    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c53cba8d-868b9cc99-vsgdq"]
Nov 03 08:09:54.961227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.961206    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:54.968773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.968748    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"]
Nov 03 08:09:54.971746 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:54.971728    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c53cba8d-875c5bd85-wdkjr"]
Nov 03 08:09:55.461072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:55.460283    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"]
Nov 03 08:09:55.461072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:55.460515    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62" podUID="9a617889-a041-4f62-aa92-efca6335917a" containerName="maas-api" containerID="cri-o://a7d102a4fbb4f200f00e655003862dd66407b2dcfe9ca7e5ccfceb6f3c772d51" gracePeriod=30
Nov 03 08:09:55.518604 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:55.518534    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:55.692005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:55.691917    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:55.973032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:55.972800    6430 generic.go:356] "Generic (PLEG): container finished" podID="9a617889-a041-4f62-aa92-efca6335917a" containerID="a7d102a4fbb4f200f00e655003862dd66407b2dcfe9ca7e5ccfceb6f3c772d51" exitCode=0
Nov 03 08:09:55.973032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:55.972950    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62" event={"ID":"9a617889-a041-4f62-aa92-efca6335917a","Type":"ContainerDied","Data":"a7d102a4fbb4f200f00e655003862dd66407b2dcfe9ca7e5ccfceb6f3c772d51"}
Nov 03 08:09:56.070634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.070537    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:56.208774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.208748    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:56.257138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.256002    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a617889-a041-4f62-aa92-efca6335917a-kube-api-access-vbcwq\" (UniqueName: \"kubernetes.io/projected/9a617889-a041-4f62-aa92-efca6335917a-kube-api-access-vbcwq\") pod \"9a617889-a041-4f62-aa92-efca6335917a\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") "
Nov 03 08:09:56.257138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.256455    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-metrics-certs\") pod \"9a617889-a041-4f62-aa92-efca6335917a\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") "
Nov 03 08:09:56.257138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.256582    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-maas-api-tls\") pod \"9a617889-a041-4f62-aa92-efca6335917a\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") "
Nov 03 08:09:56.257138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.256618    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a617889-a041-4f62-aa92-efca6335917a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a617889-a041-4f62-aa92-efca6335917a-openshift-service-ca\") pod \"9a617889-a041-4f62-aa92-efca6335917a\" (UID: \"9a617889-a041-4f62-aa92-efca6335917a\") "
Nov 03 08:09:56.311662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.294252    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a617889-a041-4f62-aa92-efca6335917a-openshift-service-ca" pod "9a617889-a041-4f62-aa92-efca6335917a" (UID: "9a617889-a041-4f62-aa92-efca6335917a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:56.311662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.310430    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-metrics-certs" pod "9a617889-a041-4f62-aa92-efca6335917a" (UID: "9a617889-a041-4f62-aa92-efca6335917a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:56.311662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.311447    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:56.341102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.326992    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a617889-a041-4f62-aa92-efca6335917a-kube-api-access-vbcwq" pod "9a617889-a041-4f62-aa92-efca6335917a" (UID: "9a617889-a041-4f62-aa92-efca6335917a"). InnerVolumeSpecName "kube-api-access-vbcwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:56.343511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.341499    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-maas-api-tls" pod "9a617889-a041-4f62-aa92-efca6335917a" (UID: "9a617889-a041-4f62-aa92-efca6335917a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:56.358893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.358804    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbcwq\" (UniqueName: \"kubernetes.io/projected/9a617889-a041-4f62-aa92-efca6335917a-kube-api-access-vbcwq\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:56.358893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.358839    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:56.358893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.358854    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a617889-a041-4f62-aa92-efca6335917a-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:56.358893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.358868    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a617889-a041-4f62-aa92-efca6335917a-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:56.617112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.617011    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:56.621483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.621441    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31cb25d8-4419-4758-9a1c-36415878e948" path="/var/lib/kubelet/pods/31cb25d8-4419-4758-9a1c-36415878e948/volumes"
Nov 03 08:09:56.622044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.622027    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b005dc-4614-45bd-b172-70f28c6e8638" path="/var/lib/kubelet/pods/a8b005dc-4614-45bd-b172-70f28c6e8638/volumes"
Nov 03 08:09:56.864846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:56.864533    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:57.134085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.115166    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62" event={"ID":"9a617889-a041-4f62-aa92-efca6335917a","Type":"ContainerDied","Data":"64a0255be019c7fc6b20a0d36ffdd22d02b53e818b917f11b3d7dcd39b9e3f63"}
Nov 03 08:09:57.134085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.115239    6430 scope.go:122] "RemoveContainer" containerID="a7d102a4fbb4f200f00e655003862dd66407b2dcfe9ca7e5ccfceb6f3c772d51"
Nov 03 08:09:57.134085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.115376    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"
Nov 03 08:09:57.159871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.159729    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:57.161509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.161488    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"]
Nov 03 08:09:57.194752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.194727    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b241313-75649cc5c6-6cq62"]
Nov 03 08:09:57.316110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.314650    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:57.433837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.433804    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:57.547103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.510446    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"]
Nov 03 08:09:57.547103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.510765    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr" podUID="cacd7c89-25a0-4db6-b494-98685ae4724e" containerName="istio-proxy" containerID="cri-o://69fa83bbe0a057811fd294304b74914f0248dff581c3690f4e543cdd5913720f" gracePeriod=30
Nov 03 08:09:57.566452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.564353    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:57.965697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:57.963603    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:58.196353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:58.196306    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:58.232135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:58.231988    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk" podUID="211eaa2f-16f4-4e48-8060-7aa70039c7bd" containerName="payload-pre-processing" containerID="cri-o://891ed2df178857d88ea98290a7eff3db6f589f528ff3798c309833880c1da44c" gracePeriod=25
Nov 03 08:09:58.235274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:58.232372    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb" podUID="c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" containerName="payload-processing" containerID="cri-o://cdbad2130edadd3d9fc377d5a399d2dbda9cc8459e19f484b3f0df1b9c309057" gracePeriod=25
Nov 03 08:09:58.473375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:58.473027    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:58.524213 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:09:58.515242    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:09:58.598264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:58.597944    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:58.655369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:58.655286    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a617889-a041-4f62-aa92-efca6335917a" path="/var/lib/kubelet/pods/9a617889-a041-4f62-aa92-efca6335917a/volumes"
Nov 03 08:09:59.163218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.152005    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:59.224561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.224363    6430 generic.go:356] "Generic (PLEG): container finished" podID="c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" containerID="cdbad2130edadd3d9fc377d5a399d2dbda9cc8459e19f484b3f0df1b9c309057" exitCode=0
Nov 03 08:09:59.224561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.224478    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb" event={"ID":"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f","Type":"ContainerDied","Data":"cdbad2130edadd3d9fc377d5a399d2dbda9cc8459e19f484b3f0df1b9c309057"}
Nov 03 08:09:59.253497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.253457    6430 generic.go:356] "Generic (PLEG): container finished" podID="211eaa2f-16f4-4e48-8060-7aa70039c7bd" containerID="891ed2df178857d88ea98290a7eff3db6f589f528ff3798c309833880c1da44c" exitCode=0
Nov 03 08:09:59.253924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.253900    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk" event={"ID":"211eaa2f-16f4-4e48-8060-7aa70039c7bd","Type":"ContainerDied","Data":"891ed2df178857d88ea98290a7eff3db6f589f528ff3798c309833880c1da44c"}
Nov 03 08:09:59.254500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.254479    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"]
Nov 03 08:09:59.258655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.258623    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9a617889-a041-4f62-aa92-efca6335917a" containerName="maas-api"
Nov 03 08:09:59.258851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.258833    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a617889-a041-4f62-aa92-efca6335917a" containerName="maas-api"
Nov 03 08:09:59.258995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.258983    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a8b005dc-4614-45bd-b172-70f28c6e8638" containerName="payload-pre-processing"
Nov 03 08:09:59.259089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.259076    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b005dc-4614-45bd-b172-70f28c6e8638" containerName="payload-pre-processing"
Nov 03 08:09:59.259251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.259238    6430 cpu_manager.go:414] "containerMap: removing container" podUID="31cb25d8-4419-4758-9a1c-36415878e948" containerName="payload-processing"
Nov 03 08:09:59.259329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.259319    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31cb25d8-4419-4758-9a1c-36415878e948" containerName="payload-processing"
Nov 03 08:09:59.259810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.259793    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a617889-a041-4f62-aa92-efca6335917a" containerName="maas-api"
Nov 03 08:09:59.259945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.259934    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b005dc-4614-45bd-b172-70f28c6e8638" containerName="payload-pre-processing"
Nov 03 08:09:59.260025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.260016    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="31cb25d8-4419-4758-9a1c-36415878e948" containerName="payload-processing"
Nov 03 08:09:59.291581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.291542    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"]
Nov 03 08:09:59.291944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.291924    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.319250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.319110    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a61a48-openshift-default-dockercfg-zwvhz\""
Nov 03 08:09:59.499413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.499329    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"]
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.504889    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.511511    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-fe584a-serving-cert\""
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.523974    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czv4n\" (UniqueName: \"kubernetes.io/projected/9fe323dc-a106-4bea-a996-d0602f6c7323-kube-api-access-czv4n\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524031    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqnx\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-kube-api-access-grqnx\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524145    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-podinfo\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524180    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-data\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524209    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe323dc-a106-4bea-a996-d0602f6c7323-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524251    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524324    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-socket\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524366    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-token\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524403    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-envoy\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524452    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-credential-socket\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524478    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istiod-ca-cert\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524568    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.527088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.524616    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-certs\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.539164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.533697    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:59.625852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.603467    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:09:59.644086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.626650    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211eaa2f-16f4-4e48-8060-7aa70039c7bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eaa2f-16f4-4e48-8060-7aa70039c7bd-plugins-config-volume\") pod \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") "
Nov 03 08:09:59.644086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.626883    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211eaa2f-16f4-4e48-8060-7aa70039c7bd-kube-api-access-s2pcz\" (UniqueName: \"kubernetes.io/projected/211eaa2f-16f4-4e48-8060-7aa70039c7bd-kube-api-access-s2pcz\") pod \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\" (UID: \"211eaa2f-16f4-4e48-8060-7aa70039c7bd\") "
Nov 03 08:09:59.663087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.658200    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211eaa2f-16f4-4e48-8060-7aa70039c7bd-plugins-config-volume" pod "211eaa2f-16f4-4e48-8060-7aa70039c7bd" (UID: "211eaa2f-16f4-4e48-8060-7aa70039c7bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.689523    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czv4n\" (UniqueName: \"kubernetes.io/projected/9fe323dc-a106-4bea-a996-d0602f6c7323-kube-api-access-czv4n\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.689592    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqnx\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-kube-api-access-grqnx\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.689742    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-podinfo\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.689783    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-data\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.689811    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe323dc-a106-4bea-a996-d0602f6c7323-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.689870    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.689919    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-socket\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.690097    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-token\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.690149    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-envoy\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.730319    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-credential-socket\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.730393    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istiod-ca-cert\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.730544    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.730613    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-certs\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.730838    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eaa2f-16f4-4e48-8060-7aa70039c7bd-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.731316    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-certs\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.731808    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-data\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.732847    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe323dc-a106-4bea-a996-d0602f6c7323-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.748276    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.748577    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-socket\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.752591    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"]
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.754914    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-envoy\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.755244    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-credential-socket\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.762728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.756025    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istiod-ca-cert\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.765902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.765870    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:59.783405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.783326    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqnx\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-kube-api-access-grqnx\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.784369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.784316    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czv4n\" (UniqueName: \"kubernetes.io/projected/9fe323dc-a106-4bea-a996-d0602f6c7323-kube-api-access-czv4n\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.860089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.859619    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:09:59.871148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.869823    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-podinfo\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.871148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.870713    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-token\") pod \"e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") " pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:09:59.871148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.870891    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211eaa2f-16f4-4e48-8060-7aa70039c7bd-kube-api-access-s2pcz" pod "211eaa2f-16f4-4e48-8060-7aa70039c7bd" (UID: "211eaa2f-16f4-4e48-8060-7aa70039c7bd"). InnerVolumeSpecName "kube-api-access-s2pcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:59.884132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.883762    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:09:59.916292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.904492    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:09:59.972086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:09:59.945757    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2pcz\" (UniqueName: \"kubernetes.io/projected/211eaa2f-16f4-4e48-8060-7aa70039c7bd-kube-api-access-s2pcz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:00.048892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.048808    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-plugins-config-volume\") pod \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") "
Nov 03 08:10:00.048892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.048876    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-kube-api-access-z875t\" (UniqueName: \"kubernetes.io/projected/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-kube-api-access-z875t\") pod \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\" (UID: \"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f\") "
Nov 03 08:10:00.049344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.049316    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-plugins-config-volume" pod "c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" (UID: "c9d84714-9d1a-4e30-9039-7bdfca4e3d8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:10:00.049613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.049595    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:00.058238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.058207    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-kube-api-access-z875t" pod "c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" (UID: "c9d84714-9d1a-4e30-9039-7bdfca4e3d8f"). InnerVolumeSpecName "kube-api-access-z875t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:00.113493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.113459    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:10:00.140521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.139401    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:10:00.150864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.150835    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z875t\" (UniqueName: \"kubernetes.io/projected/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f-kube-api-access-z875t\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:00.264530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.264504    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"
Nov 03 08:10:00.264894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.264497    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb" event={"ID":"c9d84714-9d1a-4e30-9039-7bdfca4e3d8f","Type":"ContainerDied","Data":"9c37d0230420288cc4a0c77efb6ebc9dac6e9e2fcdc0a8eccc31094710d9f104"}
Nov 03 08:10:00.264894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.264671    6430 scope.go:122] "RemoveContainer" containerID="cdbad2130edadd3d9fc377d5a399d2dbda9cc8459e19f484b3f0df1b9c309057"
Nov 03 08:10:00.269095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.266895    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk" event={"ID":"211eaa2f-16f4-4e48-8060-7aa70039c7bd","Type":"ContainerDied","Data":"5b3ac5c1f0d6e13f5d5d8cb1417d49b15a151f2c284914e56eba920275dde6ab"}
Nov 03 08:10:00.269095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.266934    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"
Nov 03 08:10:00.281502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.281482    6430 scope.go:122] "RemoveContainer" containerID="891ed2df178857d88ea98290a7eff3db6f589f528ff3798c309833880c1da44c"
Nov 03 08:10:00.293288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.293256    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"]
Nov 03 08:10:00.299026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.298356    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b241313-6dd446c46f-77ltb"]
Nov 03 08:10:00.311703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.311635    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"]
Nov 03 08:10:00.319102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.317049    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b241313-9484bf9b7-p2bkk"]
Nov 03 08:10:00.494691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.494654    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"]
Nov 03 08:10:00.817333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.809022    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:10:00.817333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.809149    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:10:00.817333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.809193    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:10:00.839235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.828589    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:00.839235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.837680    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211eaa2f-16f4-4e48-8060-7aa70039c7bd" path="/var/lib/kubelet/pods/211eaa2f-16f4-4e48-8060-7aa70039c7bd/volumes"
Nov 03 08:10:00.839235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.838621    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" path="/var/lib/kubelet/pods/c9d84714-9d1a-4e30-9039-7bdfca4e3d8f/volumes"
Nov 03 08:10:00.839235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:00.839122    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"]
Nov 03 08:10:01.589131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:01.582379    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw" event={"ID":"9fe323dc-a106-4bea-a996-d0602f6c7323","Type":"ContainerStarted","Data":"0ddcdf00636454653790bcbffccfd5395180936aa4772ced784cd42e5be5aced"}
Nov 03 08:10:01.589131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:01.588825    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:01.604293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:01.592321    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" event={"ID":"8d8e8ea0-340a-4aef-8b26-ee5b79d47394","Type":"ContainerStarted","Data":"91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce"}
Nov 03 08:10:01.604293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:01.592438    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" event={"ID":"8d8e8ea0-340a-4aef-8b26-ee5b79d47394","Type":"ContainerStarted","Data":"9b12a2d58c717ab63bafdffefda22f600e904f14c9ec11f0db20801d759391df"}
Nov 03 08:10:01.653085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:01.650229    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podStartSLOduration=3.650207521 podStartE2EDuration="3.650207521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:01.648043986 +0000 UTC m=+951.150879501" watchObservedRunningTime="2026-11-03 08:10:01.650207521 +0000 UTC m=+951.153043043"
Nov 03 08:10:01.710321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:01.710286    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:01.862137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:01.853597    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:02.186218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.185967    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:10:02.249904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.249865    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:10:02.571228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.571131    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:02.744244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.743766    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw" event={"ID":"9fe323dc-a106-4bea-a996-d0602f6c7323","Type":"ContainerStarted","Data":"81b490298990097262a1891ef8043ec01942ece72b910904bcc213adef15e6b5"}
Nov 03 08:10:02.745827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.744730    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:10:02.745827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.744807    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:10:02.751107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.746765    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:10:02.764127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.763076    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw" podStartSLOduration=3.077589152 podStartE2EDuration="3.763040924s" totalImagesPullingTime="685.451772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:02.759290406 +0000 UTC m=+952.262125923" watchObservedRunningTime="2026-11-03 08:10:02.763040924 +0000 UTC m=+952.265876438"
Nov 03 08:10:02.921475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.921382    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:02.970750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.970716    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"]
Nov 03 08:10:02.971414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.971392    6430 cpu_manager.go:414] "containerMap: removing container" podUID="211eaa2f-16f4-4e48-8060-7aa70039c7bd" containerName="payload-pre-processing"
Nov 03 08:10:02.971414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.971409    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211eaa2f-16f4-4e48-8060-7aa70039c7bd" containerName="payload-pre-processing"
Nov 03 08:10:02.971608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.971457    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" containerName="payload-processing"
Nov 03 08:10:02.971608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.971462    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" containerName="payload-processing"
Nov 03 08:10:02.971608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.971554    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d84714-9d1a-4e30-9039-7bdfca4e3d8f" containerName="payload-processing"
Nov 03 08:10:02.971608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.971565    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="211eaa2f-16f4-4e48-8060-7aa70039c7bd" containerName="payload-pre-processing"
Nov 03 08:10:02.975406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.975384    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:02.981450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.981423    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7a6481\""
Nov 03 08:10:02.982140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.982120    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7a6481-dockercfg-z4vt2\""
Nov 03 08:10:02.987331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:02.987307    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"]
Nov 03 08:10:03.021739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.021701    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"]
Nov 03 08:10:03.038795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.037991    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"]
Nov 03 08:10:03.038795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.038166    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.040785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.040764    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"]
Nov 03 08:10:03.045451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.045426    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6a9d25-ltjdr"]
Nov 03 08:10:03.047843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.047797    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:03.068245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.068211    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a521420-dbc3-4449-91c1-af058118cfb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:03.187146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.150703    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27sz5\" (UniqueName: \"kubernetes.io/projected/5a521420-dbc3-4449-91c1-af058118cfb9-kube-api-access-27sz5\") pod \"payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:03.283136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.281980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c424eb-6899-4156-a468-21a2b95f28bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.283136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.282161    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sz5\" (UniqueName: \"kubernetes.io/projected/5a521420-dbc3-4449-91c1-af058118cfb9-kube-api-access-27sz5\") pod \"payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:03.283136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.282327    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkzt\" (UniqueName: \"kubernetes.io/projected/a1c424eb-6899-4156-a468-21a2b95f28bd-kube-api-access-zjkzt\") pod \"payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.283136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.282490    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a521420-dbc3-4449-91c1-af058118cfb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:03.302100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.283564    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a521420-dbc3-4449-91c1-af058118cfb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:03.397274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.384480    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c424eb-6899-4156-a468-21a2b95f28bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.397274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.384784    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkzt\" (UniqueName: \"kubernetes.io/projected/a1c424eb-6899-4156-a468-21a2b95f28bd-kube-api-access-zjkzt\") pod \"payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.397274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.386969    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c424eb-6899-4156-a468-21a2b95f28bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.459615 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.445488    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27sz5\" (UniqueName: \"kubernetes.io/projected/5a521420-dbc3-4449-91c1-af058118cfb9-kube-api-access-27sz5\") pod \"payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:03.481855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.479831    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkzt\" (UniqueName: \"kubernetes.io/projected/a1c424eb-6899-4156-a468-21a2b95f28bd-kube-api-access-zjkzt\") pod \"payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.601594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.601510    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:03.708476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.708440    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:03.728602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.728563    6430 patch_prober.go:29] interesting pod/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 08:10:03.728732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.728627    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr" podUID="cacd7c89-25a0-4db6-b494-98685ae4724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 08:10:03.762320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.762284    6430 generic.go:356] "Generic (PLEG): container finished" podID="cacd7c89-25a0-4db6-b494-98685ae4724e" containerID="69fa83bbe0a057811fd294304b74914f0248dff581c3690f4e543cdd5913720f" exitCode=0
Nov 03 08:10:03.763273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.763202    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr" event={"ID":"cacd7c89-25a0-4db6-b494-98685ae4724e","Type":"ContainerDied","Data":"69fa83bbe0a057811fd294304b74914f0248dff581c3690f4e543cdd5913720f"}
Nov 03 08:10:03.914734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.913011    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"]
Nov 03 08:10:03.914734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.913817    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:10:03.922993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:03.922970    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011146    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-token\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-token\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011217    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-socket\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011270    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-kube-api-access-qv4vw\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-kube-api-access-qv4vw\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011405    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cacd7c89-25a0-4db6-b494-98685ae4724e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacd7c89-25a0-4db6-b494-98685ae4724e-istiod-ca-cert\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011525    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-data\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011588    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-podinfo\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011684    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-envoy\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011771    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-credential-socket\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.011955    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-certs\") pod \"cacd7c89-25a0-4db6-b494-98685ae4724e\" (UID: \"cacd7c89-25a0-4db6-b494-98685ae4724e\") "
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.014165    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cacd7c89-25a0-4db6-b494-98685ae4724e-istiod-ca-cert" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.014461    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-data" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.014978    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-certs" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.015224    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-socket" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:04.016940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.015651    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-credential-socket" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:04.120367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.120339    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.120458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.120370    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.120458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.120387    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacd7c89-25a0-4db6-b494-98685ae4724e-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.120458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.120400    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.120458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.120413    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.124407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.123886    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"]
Nov 03 08:10:04.168960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.159835    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-podinfo" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:10:04.168960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.159953    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-kube-api-access-qv4vw" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "kube-api-access-qv4vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:04.168960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.160005    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-envoy" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:04.168960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.160053    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-token" pod "cacd7c89-25a0-4db6-b494-98685ae4724e" (UID: "cacd7c89-25a0-4db6-b494-98685ae4724e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:04.172816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.170045    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:04.221631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.221604    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.221631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.221633    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.221780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.221643    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.221780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.221662    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv4vw\" (UniqueName: \"kubernetes.io/projected/cacd7c89-25a0-4db6-b494-98685ae4724e-kube-api-access-qv4vw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:04.284706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.284667    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:04.618629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.618556    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40dc4f74-a658-488e-9c02-a08c20a664bb" path="/var/lib/kubelet/pods/40dc4f74-a658-488e-9c02-a08c20a664bb/volumes"
Nov 03 08:10:04.771512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.771475    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w" event={"ID":"a1c424eb-6899-4156-a468-21a2b95f28bd","Type":"ContainerStarted","Data":"4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83"}
Nov 03 08:10:04.771990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.771550    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w" event={"ID":"a1c424eb-6899-4156-a468-21a2b95f28bd","Type":"ContainerStarted","Data":"8f40e0e2d0e9c20d8a0a9a2b1064316e8f23a74b2291a360a4d51830782f0d69"}
Nov 03 08:10:04.771990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.771699    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:04.772907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.772889    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"
Nov 03 08:10:04.772983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.772922    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr" event={"ID":"cacd7c89-25a0-4db6-b494-98685ae4724e","Type":"ContainerDied","Data":"d5cf012871ee887984301e9374db0e02ba19dc90705b4a44127094b5edecfe41"}
Nov 03 08:10:04.772983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.772961    6430 scope.go:122] "RemoveContainer" containerID="69fa83bbe0a057811fd294304b74914f0248dff581c3690f4e543cdd5913720f"
Nov 03 08:10:04.774863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.774836    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8" event={"ID":"5a521420-dbc3-4449-91c1-af058118cfb9","Type":"ContainerStarted","Data":"a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3"}
Nov 03 08:10:04.774979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.774888    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8" event={"ID":"5a521420-dbc3-4449-91c1-af058118cfb9","Type":"ContainerStarted","Data":"3e21dfdf0aef7b9e1618354f86a5616e70456b0f1323812aba2273319cc6f94d"}
Nov 03 08:10:04.775111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.775091    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:04.791376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.791321    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w" podStartSLOduration=1.7913043979999999 podStartE2EDuration="1.791304398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:04.7881305 +0000 UTC m=+954.290966014" watchObservedRunningTime="2026-11-03 08:10:04.791304398 +0000 UTC m=+954.294139914"
Nov 03 08:10:04.805414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.805373    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8" podStartSLOduration=2.805358864 podStartE2EDuration="2.805358864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:04.802193299 +0000 UTC m=+954.305028816" watchObservedRunningTime="2026-11-03 08:10:04.805358864 +0000 UTC m=+954.308194378"
Nov 03 08:10:04.819098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.819053    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"]
Nov 03 08:10:04.823982 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:04.823958    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c53cba8d-openshift-default-f7f459b9-l4dzr"]
Nov 03 08:10:05.146850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.146821    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:05.427888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.427813    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:05.470242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.470215    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"]
Nov 03 08:10:05.470906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.470887    6430 cpu_manager.go:414] "containerMap: removing container" podUID="cacd7c89-25a0-4db6-b494-98685ae4724e" containerName="istio-proxy"
Nov 03 08:10:05.470980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.470909    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacd7c89-25a0-4db6-b494-98685ae4724e" containerName="istio-proxy"
Nov 03 08:10:05.471040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.471031    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacd7c89-25a0-4db6-b494-98685ae4724e" containerName="istio-proxy"
Nov 03 08:10:05.474002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.473987    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:05.480193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.480168    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a61a48\""
Nov 03 08:10:05.480525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.480506    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a61a48-dockercfg-747nf\""
Nov 03 08:10:05.485056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.485033    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"]
Nov 03 08:10:05.515629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.515598    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"]
Nov 03 08:10:05.519706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.519680    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:05.529619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.529587    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"]
Nov 03 08:10:05.633084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.631229    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjdw\" (UniqueName: \"kubernetes.io/projected/9368968e-6637-402a-998f-8075b5dddfb7-kube-api-access-2hjdw\") pod \"payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:05.633084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.631371    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88feb840-2575-4c86-a9c2-b526ad23c84c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:05.633084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.631541    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvxx\" (UniqueName: \"kubernetes.io/projected/88feb840-2575-4c86-a9c2-b526ad23c84c-kube-api-access-2fvxx\") pod \"payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:05.633084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.631605    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368968e-6637-402a-998f-8075b5dddfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:05.773100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.770501    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368968e-6637-402a-998f-8075b5dddfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:05.773100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.770772    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjdw\" (UniqueName: \"kubernetes.io/projected/9368968e-6637-402a-998f-8075b5dddfb7-kube-api-access-2hjdw\") pod \"payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:05.773100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.770916    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88feb840-2575-4c86-a9c2-b526ad23c84c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:05.773100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.771155    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvxx\" (UniqueName: \"kubernetes.io/projected/88feb840-2575-4c86-a9c2-b526ad23c84c-kube-api-access-2fvxx\") pod \"payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:05.801860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.794717    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88feb840-2575-4c86-a9c2-b526ad23c84c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:05.801860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.795017    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368968e-6637-402a-998f-8075b5dddfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:05.882087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.861807    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvxx\" (UniqueName: \"kubernetes.io/projected/88feb840-2575-4c86-a9c2-b526ad23c84c-kube-api-access-2fvxx\") pod \"payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:05.886402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.886349    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjdw\" (UniqueName: \"kubernetes.io/projected/9368968e-6637-402a-998f-8075b5dddfb7-kube-api-access-2hjdw\") pod \"payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:05.944879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:05.944841    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:06.024443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.024356    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:06.101666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.101634    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:06.257704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.257667    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:06.423995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.423924    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"]
Nov 03 08:10:06.442038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.442014    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"]
Nov 03 08:10:06.445513 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:10:06.445488    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9368968e_6637_402a_998f_8075b5dddfb7.slice/crio-e8232872f15f36b9831f29e583c2ea0d3094868164b77c381f9e8d1af3d2c827 WatchSource:0}: Error finding container e8232872f15f36b9831f29e583c2ea0d3094868164b77c381f9e8d1af3d2c827: Status 404 returned error can't find the container with id e8232872f15f36b9831f29e583c2ea0d3094868164b77c381f9e8d1af3d2c827
Nov 03 08:10:06.619257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.619220    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cacd7c89-25a0-4db6-b494-98685ae4724e" path="/var/lib/kubelet/pods/cacd7c89-25a0-4db6-b494-98685ae4724e/volumes"
Nov 03 08:10:06.880422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.880387    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" event={"ID":"9368968e-6637-402a-998f-8075b5dddfb7","Type":"ContainerStarted","Data":"0b1899b5450039c9bef7334bc3852f086612f51176e5a7b75f8ab87e74e02bf8"}
Nov 03 08:10:06.880891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.880455    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:06.880891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.880477    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" event={"ID":"9368968e-6637-402a-998f-8075b5dddfb7","Type":"ContainerStarted","Data":"e8232872f15f36b9831f29e583c2ea0d3094868164b77c381f9e8d1af3d2c827"}
Nov 03 08:10:06.882457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.882421    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" event={"ID":"88feb840-2575-4c86-a9c2-b526ad23c84c","Type":"ContainerStarted","Data":"de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c"}
Nov 03 08:10:06.882598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.882477    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:06.882598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.882500    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" event={"ID":"88feb840-2575-4c86-a9c2-b526ad23c84c","Type":"ContainerStarted","Data":"b015733f24707da67b86e1e90df96eb21925216f3fc41ac3a790a19691a647d8"}
Nov 03 08:10:06.898503 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.898455    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" podStartSLOduration=1.89844079 podStartE2EDuration="1.89844079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:06.895020635 +0000 UTC m=+956.397856149" watchObservedRunningTime="2026-11-03 08:10:06.89844079 +0000 UTC m=+956.401276304"
Nov 03 08:10:06.909350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.909306    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" podStartSLOduration=1.9092939279999999 podStartE2EDuration="1.909293928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:06.908820531 +0000 UTC m=+956.411656047" watchObservedRunningTime="2026-11-03 08:10:06.909293928 +0000 UTC m=+956.412129442"
Nov 03 08:10:06.966181 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:06.966145    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:07.718258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:07.718227    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:07.843431 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:07.843390    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:07.965287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:07.965259    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:08.561852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:08.561818    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:08.768750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:08.768722    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:10:09.338164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:09.338136    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:10.423082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:10.423039    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:10.565159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:10.564541    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:11.127144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:11.126851    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:11.569196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:11.569164    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:12.226457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:12.226423    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:12.336140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:12.328719    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:13.006280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:13.006249    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:13.236989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:13.236956    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:13.481552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:13.481507    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:13.613092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:13.612309    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:14.199448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:14.199409    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:14.322728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:14.322694    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:14.549792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:14.549598    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:15.165625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:15.165589    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:15.877367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:15.877340    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:10:15.878536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:15.878511    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:10:16.237867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:16.237830    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:16.419395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:16.419368    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:17.154364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:17.154317    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:17.279016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:17.278986    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:17.893664 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:17.893633    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:10:17.893841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:17.893716    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:10:18.240643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.240616    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:18.492352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.492275    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:18.533160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.533130    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"]
Nov 03 08:10:18.553189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.553129    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:18.554126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.554098    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"]
Nov 03 08:10:18.561887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.561860    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb887217-dockercfg-6xm2d\""
Nov 03 08:10:18.562018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.561909    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb887217\""
Nov 03 08:10:18.571120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.571095    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"]
Nov 03 08:10:18.574584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.574559    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:18.584623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.584583    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"]
Nov 03 08:10:18.614204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.614176    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:18.619753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.619720    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztz4\" (UniqueName: \"kubernetes.io/projected/4c997a69-3894-482c-9157-245f4fc76a7f-kube-api-access-vztz4\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:18.619891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.619866    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef0b8fb-90fb-4b71-995f-4f3535c78445-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:18.619958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.619901    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c997a69-3894-482c-9157-245f4fc76a7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:18.619958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.619955    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtcfm\" (UniqueName: \"kubernetes.io/projected/5ef0b8fb-90fb-4b71-995f-4f3535c78445-kube-api-access-xtcfm\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:18.738087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.734324    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztz4\" (UniqueName: \"kubernetes.io/projected/4c997a69-3894-482c-9157-245f4fc76a7f-kube-api-access-vztz4\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:18.738087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.734613    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef0b8fb-90fb-4b71-995f-4f3535c78445-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:18.738087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.734677    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c997a69-3894-482c-9157-245f4fc76a7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:18.738087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.734771    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtcfm\" (UniqueName: \"kubernetes.io/projected/5ef0b8fb-90fb-4b71-995f-4f3535c78445-kube-api-access-xtcfm\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:18.738087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.736517    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef0b8fb-90fb-4b71-995f-4f3535c78445-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:18.865143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.813555    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c997a69-3894-482c-9157-245f4fc76a7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:18.896044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.895984    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztz4\" (UniqueName: \"kubernetes.io/projected/4c997a69-3894-482c-9157-245f4fc76a7f-kube-api-access-vztz4\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:18.924123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:18.915434    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtcfm\" (UniqueName: \"kubernetes.io/projected/5ef0b8fb-90fb-4b71-995f-4f3535c78445-kube-api-access-xtcfm\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:19.172421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:19.172346    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:19.191781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:19.191750    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:19.247299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:19.245939    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:19.460107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:19.460077    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"]
Nov 03 08:10:19.461586 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:10:19.461555    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c997a69_3894_482c_9157_245f4fc76a7f.slice/crio-0ea7215ae6419e8a080c4f0d4035d973744f9693f02d856ba23d7dade55bc0e5 WatchSource:0}: Error finding container 0ea7215ae6419e8a080c4f0d4035d973744f9693f02d856ba23d7dade55bc0e5: Status 404 returned error can't find the container with id 0ea7215ae6419e8a080c4f0d4035d973744f9693f02d856ba23d7dade55bc0e5
Nov 03 08:10:19.570552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:19.570525    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"]
Nov 03 08:10:19.571813 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:10:19.571788    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef0b8fb_90fb_4b71_995f_4f3535c78445.slice/crio-0652e0aadb31bf6ce6b82d599e34116820b249ed8100e8c76f9d809b2a56a224 WatchSource:0}: Error finding container 0652e0aadb31bf6ce6b82d599e34116820b249ed8100e8c76f9d809b2a56a224: Status 404 returned error can't find the container with id 0652e0aadb31bf6ce6b82d599e34116820b249ed8100e8c76f9d809b2a56a224
Nov 03 08:10:20.062980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.062931    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:20.218088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.215923    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" event={"ID":"5ef0b8fb-90fb-4b71-995f-4f3535c78445","Type":"ContainerStarted","Data":"2e82777dec3e1b7393f21f2642559efd2ef988753d210d18ed3f561d4f578047"}
Nov 03 08:10:20.218088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.216012    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" event={"ID":"5ef0b8fb-90fb-4b71-995f-4f3535c78445","Type":"ContainerStarted","Data":"0652e0aadb31bf6ce6b82d599e34116820b249ed8100e8c76f9d809b2a56a224"}
Nov 03 08:10:20.218088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.217630    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:20.223888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.223411    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" event={"ID":"4c997a69-3894-482c-9157-245f4fc76a7f","Type":"ContainerStarted","Data":"5dee8730e2809cdfb41bd08adf2acd07470c9375563182bf7912d4df2a8d8364"}
Nov 03 08:10:20.223888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.223474    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" event={"ID":"4c997a69-3894-482c-9157-245f4fc76a7f","Type":"ContainerStarted","Data":"0ea7215ae6419e8a080c4f0d4035d973744f9693f02d856ba23d7dade55bc0e5"}
Nov 03 08:10:20.223888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.223647    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:20.236608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.236559    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" podStartSLOduration=2.236544912 podStartE2EDuration="2.236544912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:20.233410961 +0000 UTC m=+969.736246488" watchObservedRunningTime="2026-11-03 08:10:20.236544912 +0000 UTC m=+969.739380425"
Nov 03 08:10:20.250011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.249968    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" podStartSLOduration=2.249955325 podStartE2EDuration="2.249955325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:20.247186742 +0000 UTC m=+969.750022257" watchObservedRunningTime="2026-11-03 08:10:20.249955325 +0000 UTC m=+969.752790835"
Nov 03 08:10:20.926449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:20.926405    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:21.759968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:21.759938    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:23.152116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:23.152047    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:23.291087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:23.288708    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:24.162159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:24.162126    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:24.938169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:24.938138    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:25.639010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:25.638972    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:26.739728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:26.739674    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:26.929091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:26.925592    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:27.217098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:27.213562    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:27.581964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:27.581868    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:27.727580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:27.727525    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:27.846780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:27.846698    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:28.069587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:28.069553    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:28.552404 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:10:28.552353    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:10:28.676363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:28.673577    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:28.953214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:28.953168    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:28.981213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:28.981167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:29.063387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:29.063354    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:29.470566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:29.470519    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:29.612018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:29.611988    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:30.126853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:30.111911    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:30.400211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:30.400091    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:31.025372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:31.025324    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:31.243195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:31.243170    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:10:31.524632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:31.524591    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:31.642911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:31.642874    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:32.041827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.041793    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:32.292135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.292035    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:10:32.742000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.741969    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:32.792442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.792411    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"]
Nov 03 08:10:32.796938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.796917    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:32.804995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.804810    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-e81ace\""
Nov 03 08:10:32.804995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.804954    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-e81ace-dockercfg-zlbpk\""
Nov 03 08:10:32.806744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.806724    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"]
Nov 03 08:10:32.831908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.831879    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"]
Nov 03 08:10:32.837453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.837159    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:32.843106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.842379    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"]
Nov 03 08:10:32.871374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.871346    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:32.889971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.887269    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2lm\" (UniqueName: \"kubernetes.io/projected/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-kube-api-access-tb2lm\") pod \"payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:32.889971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.887455    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:32.999133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.995461    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be030d14-64df-4b28-8224-145b22b369a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:32.999133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.995592    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2lm\" (UniqueName: \"kubernetes.io/projected/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-kube-api-access-tb2lm\") pod \"payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:32.999133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.995682    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llqtn\" (UniqueName: \"kubernetes.io/projected/be030d14-64df-4b28-8224-145b22b369a5-kube-api-access-llqtn\") pod \"payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:32.999133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.995808    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:32.999133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.998188    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:32.999133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:32.998450    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:33.045556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.045528    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2lm\" (UniqueName: \"kubernetes.io/projected/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-kube-api-access-tb2lm\") pod \"payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:33.144087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.139337    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llqtn\" (UniqueName: \"kubernetes.io/projected/be030d14-64df-4b28-8224-145b22b369a5-kube-api-access-llqtn\") pod \"payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:33.144087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.139953    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be030d14-64df-4b28-8224-145b22b369a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:33.171090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.170992    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:33.171090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.171043    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be030d14-64df-4b28-8224-145b22b369a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:33.527584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.527547    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llqtn\" (UniqueName: \"kubernetes.io/projected/be030d14-64df-4b28-8224-145b22b369a5-kube-api-access-llqtn\") pod \"payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:33.724426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.724395    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"]
Nov 03 08:10:33.726495 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:10:33.726468    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9ad8b7_51ea_4317_9ec1_2d9b9cb0ac9b.slice/crio-df38018cf85ac5cf4ec7acf34e7874e4e9274afe04719b8dd4d03dec7640be74 WatchSource:0}: Error finding container df38018cf85ac5cf4ec7acf34e7874e4e9274afe04719b8dd4d03dec7640be74: Status 404 returned error can't find the container with id df38018cf85ac5cf4ec7acf34e7874e4e9274afe04719b8dd4d03dec7640be74
Nov 03 08:10:33.764266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:33.764243    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:34.033980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.033894    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:34.073086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.051735    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"]
Nov 03 08:10:34.241075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.241025    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:34.366275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.366196    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:34.564422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.564382    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks" event={"ID":"be030d14-64df-4b28-8224-145b22b369a5","Type":"ContainerStarted","Data":"3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75"}
Nov 03 08:10:34.564851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.564453    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks" event={"ID":"be030d14-64df-4b28-8224-145b22b369a5","Type":"ContainerStarted","Data":"2d5e524c3ba81394ca8bf66c9ce584f34e97f66bd49287921f6271d446db8746"}
Nov 03 08:10:34.564851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.564502    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:34.566174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.566145    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw" event={"ID":"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b","Type":"ContainerStarted","Data":"b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24"}
Nov 03 08:10:34.566301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.566188    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw" event={"ID":"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b","Type":"ContainerStarted","Data":"df38018cf85ac5cf4ec7acf34e7874e4e9274afe04719b8dd4d03dec7640be74"}
Nov 03 08:10:34.566301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.566230    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:34.581815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.581769    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks" podStartSLOduration=2.581754644 podStartE2EDuration="2.581754644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:34.580390813 +0000 UTC m=+984.083226321" watchObservedRunningTime="2026-11-03 08:10:34.581754644 +0000 UTC m=+984.084590173"
Nov 03 08:10:34.596891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.596844    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw" podStartSLOduration=2.596829499 podStartE2EDuration="2.596829499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:34.595817092 +0000 UTC m=+984.098652605" watchObservedRunningTime="2026-11-03 08:10:34.596829499 +0000 UTC m=+984.099665016"
Nov 03 08:10:34.965671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:34.965641    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:35.487674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.487636    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:35.576611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.576584    6430 generic.go:356] "Generic (PLEG): container finished" podID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerID="14bfff959d007fa00cca4446875837aab2b419670c3e949a9b20f5983f20f76f" exitCode=0
Nov 03 08:10:35.577102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.576660    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" event={"ID":"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda","Type":"ContainerDied","Data":"14bfff959d007fa00cca4446875837aab2b419670c3e949a9b20f5983f20f76f"}
Nov 03 08:10:35.577102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.577032    6430 scope.go:122] "RemoveContainer" containerID="14bfff959d007fa00cca4446875837aab2b419670c3e949a9b20f5983f20f76f"
Nov 03 08:10:35.578551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.578535    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"]
Nov 03 08:10:35.578621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.578556    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:10:35.578621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.578612    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:10:35.578685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.578646    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:10:35.582864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.582850    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:35.594583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.594564    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"]
Nov 03 08:10:35.630855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.629253    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:35.643079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.640707    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-d6a8bf\""
Nov 03 08:10:35.643079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.640772    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-d6a8bf-dockercfg-v78ql\""
Nov 03 08:10:35.676107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.676048    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmbp\" (UniqueName: \"kubernetes.io/projected/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-kube-api-access-tpmbp\") pod \"payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:35.676447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.676425    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:35.774556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.774479    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:35.778409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.778386    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmbp\" (UniqueName: \"kubernetes.io/projected/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-kube-api-access-tpmbp\") pod \"payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:35.778519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.778427    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:35.779141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.779124    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:35.786531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.786482    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmbp\" (UniqueName: \"kubernetes.io/projected/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-kube-api-access-tpmbp\") pod \"payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:35.892443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.892353    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:10:35.941120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:35.938812    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:36.547946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:36.547921    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"]
Nov 03 08:10:36.551576 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:10:36.551528    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fc0cdf_c4e1_4cd5_addb_ab4a4c6646ca.slice/crio-035ac6d39f605f2c485607a93f8e3bcc76500058cb8cdd04f576cc8b499e6b61 WatchSource:0}: Error finding container 035ac6d39f605f2c485607a93f8e3bcc76500058cb8cdd04f576cc8b499e6b61: Status 404 returned error can't find the container with id 035ac6d39f605f2c485607a93f8e3bcc76500058cb8cdd04f576cc8b499e6b61
Nov 03 08:10:36.586686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:36.586660    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" event={"ID":"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda","Type":"ContainerStarted","Data":"ac5d2f23d2e52733f714494c9c02dc13eea515b49ae6e525b9a82b1d4d0aa8f0"}
Nov 03 08:10:36.588225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:36.588197    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" event={"ID":"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca","Type":"ContainerStarted","Data":"035ac6d39f605f2c485607a93f8e3bcc76500058cb8cdd04f576cc8b499e6b61"}
Nov 03 08:10:36.637619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:36.637580    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:37.196114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:37.191159    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:10:37.196114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:37.191292    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 08:10:37.196114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:37.191351    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 08:10:37.347495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:37.347457    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:37.596639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:37.596547    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" event={"ID":"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca","Type":"ContainerStarted","Data":"267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033"}
Nov 03 08:10:37.597010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:37.596642    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:37.637280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:37.637223    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" podStartSLOduration=2.637207374 podStartE2EDuration="2.637207374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:37.633609504 +0000 UTC m=+987.136445030" watchObservedRunningTime="2026-11-03 08:10:37.637207374 +0000 UTC m=+987.140043090"
Nov 03 08:10:38.046809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:38.046777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:38.197094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:38.191231    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 08:10:38.197094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:38.191324    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 08:10:38.760177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:38.760145    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:39.187697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:39.187665    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 08:10:39.187884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:39.187732    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 08:10:39.447906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:39.447827    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:40.193085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:40.188544    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 08:10:40.193085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:40.188607    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 08:10:40.458143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:40.455234    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:41.188689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:41.188655    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 08:10:41.188856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:41.188718    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 08:10:41.328345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:41.328308    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:42.047581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.047546    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"]
Nov 03 08:10:42.053128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.053108    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.062224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.062198    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-e81ace-serving-cert\""
Nov 03 08:10:42.062692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.062659    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"]
Nov 03 08:10:42.068012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.067761    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26htz\" (UniqueName: \"kubernetes.io/projected/fcadc141-2f8d-4474-81cd-9aeac02c02aa-kube-api-access-26htz\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.068012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.067901    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcadc141-2f8d-4474-81cd-9aeac02c02aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.068012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.067947    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.068189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.068166    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.123491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.123437    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:42.169411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.169373    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcadc141-2f8d-4474-81cd-9aeac02c02aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.169585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.169427    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.169585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.169569    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.169665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.169623    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26htz\" (UniqueName: \"kubernetes.io/projected/fcadc141-2f8d-4474-81cd-9aeac02c02aa-kube-api-access-26htz\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.170155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.170131    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcadc141-2f8d-4474-81cd-9aeac02c02aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.172468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.172443    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.172577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.172468    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.176516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.176489    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26htz\" (UniqueName: \"kubernetes.io/projected/fcadc141-2f8d-4474-81cd-9aeac02c02aa-kube-api-access-26htz\") pod \"maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:42.188666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.188633    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 08:10:42.188766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.188719    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 08:10:42.242228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.242188    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:42.348239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.348139    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:42.379673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:42.379621    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:43.085640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.038466    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"]
Nov 03 08:10:43.188001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.187970    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:43.193693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.193670    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:10:43.194748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.194721    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:10:43.217359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.213205    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:10:43.705073 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.705030    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4" event={"ID":"fcadc141-2f8d-4474-81cd-9aeac02c02aa","Type":"ContainerStarted","Data":"e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf"}
Nov 03 08:10:43.705576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.705173    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4" event={"ID":"fcadc141-2f8d-4474-81cd-9aeac02c02aa","Type":"ContainerStarted","Data":"dd3a0a603b215174e4ad9bf643756b2dbdcfeebd2b4f9f9acb3b514053a539b5"}
Nov 03 08:10:43.720611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:43.720556    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4" podStartSLOduration=1.2519056370000001 podStartE2EDuration="1.720540649s" totalImagesPullingTime="468.635012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:43.720324558 +0000 UTC m=+993.223160072" watchObservedRunningTime="2026-11-03 08:10:43.720540649 +0000 UTC m=+993.223376169"
Nov 03 08:10:44.025446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.025412    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:44.698228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.698195    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"]
Nov 03 08:10:44.702835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.702815    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.706456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.706342    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb887217\"/\"kube-root-ca.crt\""
Nov 03 08:10:44.706456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.706404    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb887217\"/\"openshift-service-ca.crt\""
Nov 03 08:10:44.709395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.709272    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb887217\"/\"default-dockercfg-lsslm\""
Nov 03 08:10:44.709567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.709548    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb887217\"/\"e2e-lifecycle-model-cb887217-kserve-self-signed-certs\""
Nov 03 08:10:44.714755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.712521    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"]
Nov 03 08:10:44.719418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.718148    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:44.776325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.776277    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:44.780187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.780158    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-home\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.780349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.780206    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a29667-3653-485c-8cac-3471c5a15ba9-tls-certs\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.780469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.780442    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-dshm\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.780573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.780555    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-tmp-dir\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.780629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.780611    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2sxs\" (UniqueName: \"kubernetes.io/projected/13a29667-3653-485c-8cac-3471c5a15ba9-kube-api-access-c2sxs\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.780815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.780791    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-model-cache\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.881757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.881725    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-tmp-dir\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.881946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.881785    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2sxs\" (UniqueName: \"kubernetes.io/projected/13a29667-3653-485c-8cac-3471c5a15ba9-kube-api-access-c2sxs\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.881946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.881850    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-model-cache\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.882136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.882074    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-home\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.882211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.882136    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a29667-3653-485c-8cac-3471c5a15ba9-tls-certs\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.882263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.882231    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-tmp-dir\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.882330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.882307    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-model-cache\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.882449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.882421    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-home\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.882659 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.882639    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-dshm\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.884650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.884628    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a29667-3653-485c-8cac-3471c5a15ba9-tls-certs\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.884811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.884792    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-dshm\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.888990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.888971    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2sxs\" (UniqueName: \"kubernetes.io/projected/13a29667-3653-485c-8cac-3471c5a15ba9-kube-api-access-c2sxs\") pod \"e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") " pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:44.930121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:44.930084    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:45.043147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.039365    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:45.574350 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:10:45.574303    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13a29667_3653_485c_8cac_3471c5a15ba9.slice/crio-a063d7b79ee9a4a5e521ef53aaf8d1f9b3ec95ce5b8ba2c5f43060213cce534a WatchSource:0}: Error finding container a063d7b79ee9a4a5e521ef53aaf8d1f9b3ec95ce5b8ba2c5f43060213cce534a: Status 404 returned error can't find the container with id a063d7b79ee9a4a5e521ef53aaf8d1f9b3ec95ce5b8ba2c5f43060213cce534a
Nov 03 08:10:45.574350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.574332    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"]
Nov 03 08:10:45.581911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.581891    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:10:45.583859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.583840    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:10:45.744797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.744603    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" event={"ID":"13a29667-3653-485c-8cac-3471c5a15ba9","Type":"ContainerStarted","Data":"03bafd0fe4b55e56976f26ea8e2097648596b3fdaee1e84a9d4540902cccfd23"}
Nov 03 08:10:45.744797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.744641    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" event={"ID":"13a29667-3653-485c-8cac-3471c5a15ba9","Type":"ContainerStarted","Data":"a063d7b79ee9a4a5e521ef53aaf8d1f9b3ec95ce5b8ba2c5f43060213cce534a"}
Nov 03 08:10:45.767708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.767646    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" podStartSLOduration=1.767624413 podStartE2EDuration="1.767624413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:45.767321073 +0000 UTC m=+995.270156586" watchObservedRunningTime="2026-11-03 08:10:45.767624413 +0000 UTC m=+995.270459930"
Nov 03 08:10:45.825605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.825573    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:45.933769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:45.933738    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:46.047242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:46.047207    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:46.777471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:46.777442    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:47.544760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:47.544721    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:48.434684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:48.434651    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:48.633813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:48.633784    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:10:49.224035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:49.223987    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:50.111257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:50.111223    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:50.750594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:50.750566    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:10:50.841773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:50.841742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:51.738249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:51.738221    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:52.518624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:52.518593    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:53.656254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:53.656225    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:55.046094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:55.045603    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:55.094038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:55.090607    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:55.655089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:55.637747    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:55.987732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:55.987695    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:56.095521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:56.095482    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:57.973091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:57.972919    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.022752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:58.022163    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.142501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:58.139009    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:10:58.157849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:58.157809    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:58.327146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:58.325885    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.569295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:58.546533    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:10:58.583159 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:10:58.570980    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:10:58.986085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:58.985486    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.032689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.000408    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.082346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.082302    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:10:59.186644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.186590    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.188243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.188207    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.198407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.198379    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:59.199583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.199562    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:10:59.200679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.200664    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:10:59.204853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.204830    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:10:59.276202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.276105    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded"
Nov 03 08:10:59.372049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.372006    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.397095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.372507    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:10:59.702205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.702166    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.786126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.786052    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:10:59.854274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:10:59.854226    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:11:00.041934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:00.041848    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:11:00.051475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:00.051428    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:11:00.164090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:00.138197    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:11:00.245086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:00.243029    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:11:00.245086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:00.243674    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:11:00.500088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:00.496242    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:11:00.774079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:00.773965    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:01.729154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:01.729116    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:01.867454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:01.867350    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:02.873752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:02.873691    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:03.794087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:03.792500    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:11:03.794087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:03.792564    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:11:03.804329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:03.804278    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:04.433889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:04.433849    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:04.547435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:04.547403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:05.244402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.244366    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:05.335336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.335292    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6985-e9d00c"]
Nov 03 08:11:05.363894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.362860    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:05.364395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.364369    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-e9d00c"]
Nov 03 08:11:05.369946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.369893    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfklg\""
Nov 03 08:11:05.403351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.403314    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98q2v\" (UniqueName: \"kubernetes.io/projected/114df8fe-db5b-4a5a-bfd6-68e954ece803-kube-api-access-98q2v\") pod \"test-curl-6985-e9d00c\" (UID: \"114df8fe-db5b-4a5a-bfd6-68e954ece803\") " pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:05.504922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.504843    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98q2v\" (UniqueName: \"kubernetes.io/projected/114df8fe-db5b-4a5a-bfd6-68e954ece803-kube-api-access-98q2v\") pod \"test-curl-6985-e9d00c\" (UID: \"114df8fe-db5b-4a5a-bfd6-68e954ece803\") " pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:05.512610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.512582    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98q2v\" (UniqueName: \"kubernetes.io/projected/114df8fe-db5b-4a5a-bfd6-68e954ece803-kube-api-access-98q2v\") pod \"test-curl-6985-e9d00c\" (UID: \"114df8fe-db5b-4a5a-bfd6-68e954ece803\") " pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:05.735632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:05.735602    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:06.023604 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.023573    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-e9d00c"]
Nov 03 08:11:06.026269 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:11:06.026237    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114df8fe_db5b_4a5a_bfd6_68e954ece803.slice/crio-3cf42766d07be771e9cf22b337c0ea2e2639b149505ded9ac41e76a2e0938995 WatchSource:0}: Error finding container 3cf42766d07be771e9cf22b337c0ea2e2639b149505ded9ac41e76a2e0938995: Status 404 returned error can't find the container with id 3cf42766d07be771e9cf22b337c0ea2e2639b149505ded9ac41e76a2e0938995
Nov 03 08:11:06.109056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.109025    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:06.682493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.682458    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:06.687974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.687939    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-e9d00c" event={"ID":"114df8fe-db5b-4a5a-bfd6-68e954ece803","Type":"ContainerStarted","Data":"cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4"}
Nov 03 08:11:06.688140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.687993    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-e9d00c" event={"ID":"114df8fe-db5b-4a5a-bfd6-68e954ece803","Type":"ContainerStarted","Data":"3cf42766d07be771e9cf22b337c0ea2e2639b149505ded9ac41e76a2e0938995"}
Nov 03 08:11:06.705924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.705871    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6985-e9d00c" podStartSLOduration=1.705854958 podStartE2EDuration="1.705854958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:06.701917375 +0000 UTC m=+1016.204752890" watchObservedRunningTime="2026-11-03 08:11:06.705854958 +0000 UTC m=+1016.208690474"
Nov 03 08:11:06.789376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.789346    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:06.906952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:06.906912    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:07.059133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.045314    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:07.335723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.335644    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6985-e9d00c"]
Nov 03 08:11:07.339598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.339570    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6985-e9d00c"]
Nov 03 08:11:07.486846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.486806    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6985-eb56da"]
Nov 03 08:11:07.487534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.487518    6430 cpu_manager.go:414] "containerMap: removing container" podUID="114df8fe-db5b-4a5a-bfd6-68e954ece803" containerName="test-curl-6985-e9d00c"
Nov 03 08:11:07.487595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.487536    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114df8fe-db5b-4a5a-bfd6-68e954ece803" containerName="test-curl-6985-e9d00c"
Nov 03 08:11:07.487682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.487672    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="114df8fe-db5b-4a5a-bfd6-68e954ece803" containerName="test-curl-6985-e9d00c"
Nov 03 08:11:07.491261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.491242    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:07.495300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.495280    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-eb56da"]
Nov 03 08:11:07.546010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.545987    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ljj\" (UniqueName: \"kubernetes.io/projected/9312cfbd-9f48-4452-a6f1-1e71af7a0100-kube-api-access-l5ljj\") pod \"test-curl-6985-eb56da\" (UID: \"9312cfbd-9f48-4452-a6f1-1e71af7a0100\") " pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:07.647702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.647631    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ljj\" (UniqueName: \"kubernetes.io/projected/9312cfbd-9f48-4452-a6f1-1e71af7a0100-kube-api-access-l5ljj\") pod \"test-curl-6985-eb56da\" (UID: \"9312cfbd-9f48-4452-a6f1-1e71af7a0100\") " pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:07.655369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.655342    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ljj\" (UniqueName: \"kubernetes.io/projected/9312cfbd-9f48-4452-a6f1-1e71af7a0100-kube-api-access-l5ljj\") pod \"test-curl-6985-eb56da\" (UID: \"9312cfbd-9f48-4452-a6f1-1e71af7a0100\") " pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:07.697436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.697412    6430 generic.go:356] "Generic (PLEG): container finished" podID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerID="91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce" exitCode=0
Nov 03 08:11:07.697787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.697488    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" event={"ID":"8d8e8ea0-340a-4aef-8b26-ee5b79d47394","Type":"ContainerDied","Data":"91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce"}
Nov 03 08:11:07.697995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.697977    6430 scope.go:122] "RemoveContainer" containerID="91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce"
Nov 03 08:11:07.699654 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.699630    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:11:07.699722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.699696    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:11:07.699758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.699726    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:11:07.797380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.797349    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:07.824442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.824401    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:07.957384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:07.950098    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:08.144285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.136860    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:08.343275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.343240    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-eb56da"]
Nov 03 08:11:08.706600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.706564    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" event={"ID":"8d8e8ea0-340a-4aef-8b26-ee5b79d47394","Type":"ContainerStarted","Data":"13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e"}
Nov 03 08:11:08.708316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.708292    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-eb56da" event={"ID":"9312cfbd-9f48-4452-a6f1-1e71af7a0100","Type":"ContainerStarted","Data":"34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207"}
Nov 03 08:11:08.708446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.708320    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-eb56da" event={"ID":"9312cfbd-9f48-4452-a6f1-1e71af7a0100","Type":"ContainerStarted","Data":"e3c2c3695208657287be53fda5558e3bb7271b2b8b338de89c51b09d62e20d1a"}
Nov 03 08:11:08.708446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.708376    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6985-e9d00c" podUID="114df8fe-db5b-4a5a-bfd6-68e954ece803" containerName="test-curl-6985-e9d00c" containerID="cri-o://cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4" gracePeriod=2
Nov 03 08:11:08.739190 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.739142    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6985-eb56da" podStartSLOduration=1.739126229 podStartE2EDuration="1.739126229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:08.737540344 +0000 UTC m=+1018.240375859" watchObservedRunningTime="2026-11-03 08:11:08.739126229 +0000 UTC m=+1018.241961745"
Nov 03 08:11:08.854597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:08.854554    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:09.140812 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.140725    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:11:09.141983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.141952    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 08:11:09.142122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.142007    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 08:11:09.438722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.438613    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6985-eb56da"]
Nov 03 08:11:09.442755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.442724    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6985-eb56da"]
Nov 03 08:11:09.586372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.586341    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6985-3fd5e5"]
Nov 03 08:11:09.587146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.587125    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9312cfbd-9f48-4452-a6f1-1e71af7a0100" containerName="test-curl-6985-eb56da"
Nov 03 08:11:09.587146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.587145    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9312cfbd-9f48-4452-a6f1-1e71af7a0100" containerName="test-curl-6985-eb56da"
Nov 03 08:11:09.587378 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.587365    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9312cfbd-9f48-4452-a6f1-1e71af7a0100" containerName="test-curl-6985-eb56da"
Nov 03 08:11:09.591099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.591052    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:09.596216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.596192    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-3fd5e5"]
Nov 03 08:11:09.681820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.681788    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/42b37340-dcb9-4967-8dee-cf1c02ddf7cd-kube-api-access-zw6tr\") pod \"test-curl-6985-3fd5e5\" (UID: \"42b37340-dcb9-4967-8dee-cf1c02ddf7cd\") " pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:09.769988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.769910    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:09.783440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.783400    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/42b37340-dcb9-4967-8dee-cf1c02ddf7cd-kube-api-access-zw6tr\") pod \"test-curl-6985-3fd5e5\" (UID: \"42b37340-dcb9-4967-8dee-cf1c02ddf7cd\") " pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:09.790246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.790218    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/42b37340-dcb9-4967-8dee-cf1c02ddf7cd-kube-api-access-zw6tr\") pod \"test-curl-6985-3fd5e5\" (UID: \"42b37340-dcb9-4967-8dee-cf1c02ddf7cd\") " pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:09.952983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:09.947331    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:10.140843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.140809    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 08:11:10.141015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.140862    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 08:11:10.324376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.324343    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-3fd5e5"]
Nov 03 08:11:10.328443 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:11:10.328370    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42b37340_dcb9_4967_8dee_cf1c02ddf7cd.slice/crio-eae211f437d6cf0d57ae145f6e0f21237f7a376dd53737dc1d59c0f1332935ec WatchSource:0}: Error finding container eae211f437d6cf0d57ae145f6e0f21237f7a376dd53737dc1d59c0f1332935ec: Status 404 returned error can't find the container with id eae211f437d6cf0d57ae145f6e0f21237f7a376dd53737dc1d59c0f1332935ec
Nov 03 08:11:10.663926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.663845    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:10.804687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.804646    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6985-eb56da" podUID="9312cfbd-9f48-4452-a6f1-1e71af7a0100" containerName="test-curl-6985-eb56da" containerID="cri-o://34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207" gracePeriod=2
Nov 03 08:11:10.810719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.810686    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-3fd5e5" event={"ID":"42b37340-dcb9-4967-8dee-cf1c02ddf7cd","Type":"ContainerStarted","Data":"a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484"}
Nov 03 08:11:10.810933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.810917    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-3fd5e5" event={"ID":"42b37340-dcb9-4967-8dee-cf1c02ddf7cd","Type":"ContainerStarted","Data":"eae211f437d6cf0d57ae145f6e0f21237f7a376dd53737dc1d59c0f1332935ec"}
Nov 03 08:11:10.845761 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.845650    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6985-3fd5e5" podStartSLOduration=1.845630656 podStartE2EDuration="1.845630656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:10.837438088 +0000 UTC m=+1020.340273601" watchObservedRunningTime="2026-11-03 08:11:10.845630656 +0000 UTC m=+1020.348466173"
Nov 03 08:11:10.887157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:10.887053    6430 scope.go:122] "RemoveContainer" containerID="e8bdc8ed5469d2e67714d56727bff35da1f95a4d48cf776b0b095a4bae1df89d"
Nov 03 08:11:11.127795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.127764    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:11.140390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.140359    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 08:11:11.140521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.140411    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 08:11:11.248985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.248903    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114df8fe-db5b-4a5a-bfd6-68e954ece803-kube-api-access-98q2v\" (UniqueName: \"kubernetes.io/projected/114df8fe-db5b-4a5a-bfd6-68e954ece803-kube-api-access-98q2v\") pod \"114df8fe-db5b-4a5a-bfd6-68e954ece803\" (UID: \"114df8fe-db5b-4a5a-bfd6-68e954ece803\") "
Nov 03 08:11:11.251131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.251104    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114df8fe-db5b-4a5a-bfd6-68e954ece803-kube-api-access-98q2v" pod "114df8fe-db5b-4a5a-bfd6-68e954ece803" (UID: "114df8fe-db5b-4a5a-bfd6-68e954ece803"). InnerVolumeSpecName "kube-api-access-98q2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:11:11.351236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.351207    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98q2v\" (UniqueName: \"kubernetes.io/projected/114df8fe-db5b-4a5a-bfd6-68e954ece803-kube-api-access-98q2v\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:11.451633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.451602    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:11.498085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.495725    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6985-3fd5e5"]
Nov 03 08:11:11.722126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.722092    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6985-3fd5e5"]
Nov 03 08:11:11.722403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.722388    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6985-2630f8"]
Nov 03 08:11:11.724237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.724211    6430 cpu_manager.go:414] "containerMap: removing container" podUID="42b37340-dcb9-4967-8dee-cf1c02ddf7cd" containerName="test-curl-6985-3fd5e5"
Nov 03 08:11:11.724393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.724378    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b37340-dcb9-4967-8dee-cf1c02ddf7cd" containerName="test-curl-6985-3fd5e5"
Nov 03 08:11:11.724879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.724860    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b37340-dcb9-4967-8dee-cf1c02ddf7cd" containerName="test-curl-6985-3fd5e5"
Nov 03 08:11:11.740142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.740121    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-2630f8"]
Nov 03 08:11:11.740383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.740370    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:11.830166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.830132    6430 generic.go:356] "Generic (PLEG): container finished" podID="114df8fe-db5b-4a5a-bfd6-68e954ece803" containerID="cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4" exitCode=137
Nov 03 08:11:11.830609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.830193    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:11.830609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.830228    6430 scope.go:122] "RemoveContainer" containerID="cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4"
Nov 03 08:11:11.841085 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:11.841045    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6985-e9d00c\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="114df8fe-db5b-4a5a-bfd6-68e954ece803" pod="opendatahub/test-curl-6985-e9d00c"
Nov 03 08:11:11.841624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.841606    6430 scope.go:122] "RemoveContainer" containerID="cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4"
Nov 03 08:11:11.841892 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:11.841871    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4\": container with ID starting with cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4 not found: ID does not exist" containerID="cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4"
Nov 03 08:11:11.841947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.841917    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4"} err="failed to get container status \"cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4\": rpc error: code = NotFound desc = could not find container \"cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4\": container with ID starting with cd5ce0a79cafad7874a9ab1df509d7414587b147d20b93f6250c94161434dac4 not found: ID does not exist"
Nov 03 08:11:11.864187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.864165    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjh5k\" (UniqueName: \"kubernetes.io/projected/a3130be1-ae0e-4efe-87b7-cddcf3e564c4-kube-api-access-zjh5k\") pod \"test-curl-6985-2630f8\" (UID: \"a3130be1-ae0e-4efe-87b7-cddcf3e564c4\") " pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:11.966762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.966717    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjh5k\" (UniqueName: \"kubernetes.io/projected/a3130be1-ae0e-4efe-87b7-cddcf3e564c4-kube-api-access-zjh5k\") pod \"test-curl-6985-2630f8\" (UID: \"a3130be1-ae0e-4efe-87b7-cddcf3e564c4\") " pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:11.975713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:11.975624    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjh5k\" (UniqueName: \"kubernetes.io/projected/a3130be1-ae0e-4efe-87b7-cddcf3e564c4-kube-api-access-zjh5k\") pod \"test-curl-6985-2630f8\" (UID: \"a3130be1-ae0e-4efe-87b7-cddcf3e564c4\") " pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:12.066165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.066132    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:12.141127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.140893    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 08:11:12.141127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.140943    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 08:11:12.288998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.288894    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:12.403089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.397410    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6985-2630f8"]
Nov 03 08:11:12.420647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.420612    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:12.636632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.636602    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114df8fe-db5b-4a5a-bfd6-68e954ece803" path="/var/lib/kubelet/pods/114df8fe-db5b-4a5a-bfd6-68e954ece803/volumes"
Nov 03 08:11:12.841655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.841575    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-2630f8" event={"ID":"a3130be1-ae0e-4efe-87b7-cddcf3e564c4","Type":"ContainerStarted","Data":"9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e"}
Nov 03 08:11:12.841655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.841620    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6985-2630f8" event={"ID":"a3130be1-ae0e-4efe-87b7-cddcf3e564c4","Type":"ContainerStarted","Data":"9046abd20a4d9aa5e6f173ad769a776abd36c74fc6c180f72beadfa051d32fae"}
Nov 03 08:11:12.843213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.843184    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6985-3fd5e5" podUID="42b37340-dcb9-4967-8dee-cf1c02ddf7cd" containerName="test-curl-6985-3fd5e5" containerID="cri-o://a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484" gracePeriod=2
Nov 03 08:11:12.854411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:12.854363    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6985-2630f8" podStartSLOduration=1.854348219 podStartE2EDuration="1.854348219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:12.853045143 +0000 UTC m=+1022.355880654" watchObservedRunningTime="2026-11-03 08:11:12.854348219 +0000 UTC m=+1022.357183761"
Nov 03 08:11:13.130371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.130292    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:13.140257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.140224    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 08:11:13.140410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.140273    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 08:11:13.191079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.191038    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9312cfbd-9f48-4452-a6f1-1e71af7a0100-kube-api-access-l5ljj\" (UniqueName: \"kubernetes.io/projected/9312cfbd-9f48-4452-a6f1-1e71af7a0100-kube-api-access-l5ljj\") pod \"9312cfbd-9f48-4452-a6f1-1e71af7a0100\" (UID: \"9312cfbd-9f48-4452-a6f1-1e71af7a0100\") "
Nov 03 08:11:13.193939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.193904    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9312cfbd-9f48-4452-a6f1-1e71af7a0100-kube-api-access-l5ljj" pod "9312cfbd-9f48-4452-a6f1-1e71af7a0100" (UID: "9312cfbd-9f48-4452-a6f1-1e71af7a0100"). InnerVolumeSpecName "kube-api-access-l5ljj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:11:13.292926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.292899    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ljj\" (UniqueName: \"kubernetes.io/projected/9312cfbd-9f48-4452-a6f1-1e71af7a0100-kube-api-access-l5ljj\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:13.323786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.323752    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:13.669516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.669473    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6985-2630f8"]
Nov 03 08:11:13.673542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.673512    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6985-2630f8"]
Nov 03 08:11:13.852143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.852105    6430 generic.go:356] "Generic (PLEG): container finished" podID="9312cfbd-9f48-4452-a6f1-1e71af7a0100" containerID="34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207" exitCode=137
Nov 03 08:11:13.852569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.852199    6430 scope.go:122] "RemoveContainer" containerID="34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207"
Nov 03 08:11:13.852665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.852205    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:13.863486 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:13.863455    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6985-eb56da\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="9312cfbd-9f48-4452-a6f1-1e71af7a0100" pod="opendatahub/test-curl-6985-eb56da"
Nov 03 08:11:13.864214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.864193    6430 scope.go:122] "RemoveContainer" containerID="34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207"
Nov 03 08:11:13.864501 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:13.864482    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207\": container with ID starting with 34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207 not found: ID does not exist" containerID="34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207"
Nov 03 08:11:13.864568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:13.864511    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207"} err="failed to get container status \"34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207\": rpc error: code = NotFound desc = could not find container \"34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207\": container with ID starting with 34344267979a3602bdbe6d446775b42b32dec067716c5210a8e600bb43502207 not found: ID does not exist"
Nov 03 08:11:14.107087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:14.106620    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:14.150916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:14.143473    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 08:11:14.150916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:14.143539    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 08:11:14.546246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:14.546209    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:14.630618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:14.630572    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9312cfbd-9f48-4452-a6f1-1e71af7a0100" path="/var/lib/kubelet/pods/9312cfbd-9f48-4452-a6f1-1e71af7a0100/volumes"
Nov 03 08:11:14.865176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:14.865147    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6985-2630f8" podUID="a3130be1-ae0e-4efe-87b7-cddcf3e564c4" containerName="test-curl-6985-2630f8" containerID="cri-o://9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e" gracePeriod=2
Nov 03 08:11:14.965120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:14.964357    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:15.096657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.095613    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:15.329085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.327397    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:11:15.329085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.329015    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:11:15.345273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.330537    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:11:15.345273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.331488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:15.483049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.483010    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:15.522522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.522474    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b37340-dcb9-4967-8dee-cf1c02ddf7cd-kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/42b37340-dcb9-4967-8dee-cf1c02ddf7cd-kube-api-access-zw6tr\") pod \"42b37340-dcb9-4967-8dee-cf1c02ddf7cd\" (UID: \"42b37340-dcb9-4967-8dee-cf1c02ddf7cd\") "
Nov 03 08:11:15.588666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.586086    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b37340-dcb9-4967-8dee-cf1c02ddf7cd-kube-api-access-zw6tr" pod "42b37340-dcb9-4967-8dee-cf1c02ddf7cd" (UID: "42b37340-dcb9-4967-8dee-cf1c02ddf7cd"). InnerVolumeSpecName "kube-api-access-zw6tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:11:15.637186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.629444    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6tr\" (UniqueName: \"kubernetes.io/projected/42b37340-dcb9-4967-8dee-cf1c02ddf7cd-kube-api-access-zw6tr\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:15.873890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.873858    6430 generic.go:356] "Generic (PLEG): container finished" podID="42b37340-dcb9-4967-8dee-cf1c02ddf7cd" containerID="a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484" exitCode=137
Nov 03 08:11:15.874295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.873932    6430 scope.go:122] "RemoveContainer" containerID="a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484"
Nov 03 08:11:15.874295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.873966    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:15.882142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.882112    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:15.885019 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:15.884993    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6985-3fd5e5\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="42b37340-dcb9-4967-8dee-cf1c02ddf7cd" pod="opendatahub/test-curl-6985-3fd5e5"
Nov 03 08:11:15.885481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.885463    6430 scope.go:122] "RemoveContainer" containerID="a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484"
Nov 03 08:11:15.885746 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:15.885729    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484\": container with ID starting with a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484 not found: ID does not exist" containerID="a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484"
Nov 03 08:11:15.885815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:15.885758    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484"} err="failed to get container status \"a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484\": rpc error: code = NotFound desc = could not find container \"a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484\": container with ID starting with a3dc935a58bf923ac3493766dc053c456e59f35023b2b08a5b26b5d05cfc2484 not found: ID does not exist"
Nov 03 08:11:16.619436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:16.619405    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b37340-dcb9-4967-8dee-cf1c02ddf7cd" path="/var/lib/kubelet/pods/42b37340-dcb9-4967-8dee-cf1c02ddf7cd/volumes"
Nov 03 08:11:16.840511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:16.840480    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:17.196767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.196742    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:17.275758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.275721    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3130be1-ae0e-4efe-87b7-cddcf3e564c4-kube-api-access-zjh5k\" (UniqueName: \"kubernetes.io/projected/a3130be1-ae0e-4efe-87b7-cddcf3e564c4-kube-api-access-zjh5k\") pod \"a3130be1-ae0e-4efe-87b7-cddcf3e564c4\" (UID: \"a3130be1-ae0e-4efe-87b7-cddcf3e564c4\") "
Nov 03 08:11:17.318101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.305285    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3130be1-ae0e-4efe-87b7-cddcf3e564c4-kube-api-access-zjh5k" pod "a3130be1-ae0e-4efe-87b7-cddcf3e564c4" (UID: "a3130be1-ae0e-4efe-87b7-cddcf3e564c4"). InnerVolumeSpecName "kube-api-access-zjh5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:11:17.379083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.378539    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjh5k\" (UniqueName: \"kubernetes.io/projected/a3130be1-ae0e-4efe-87b7-cddcf3e564c4-kube-api-access-zjh5k\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:17.626984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.626950    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:17.918704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.918622    6430 generic.go:356] "Generic (PLEG): container finished" podID="a3130be1-ae0e-4efe-87b7-cddcf3e564c4" containerID="9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e" exitCode=137
Nov 03 08:11:17.918704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.918667    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:17.918704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.918694    6430 scope.go:122] "RemoveContainer" containerID="9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e"
Nov 03 08:11:17.928390 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:17.928365    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6985-2630f8\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="a3130be1-ae0e-4efe-87b7-cddcf3e564c4" pod="opendatahub/test-curl-6985-2630f8"
Nov 03 08:11:17.928858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.928844    6430 scope.go:122] "RemoveContainer" containerID="9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e"
Nov 03 08:11:17.929130 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:17.929107    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e\": container with ID starting with 9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e not found: ID does not exist" containerID="9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e"
Nov 03 08:11:17.929194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:17.929141    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e"} err="failed to get container status \"9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e\": rpc error: code = NotFound desc = could not find container \"9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e\": container with ID starting with 9be9cc0b1278ba083047e9a2dc125985f564bc8e02a35d845607c090e019ff7e not found: ID does not exist"
Nov 03 08:11:18.482640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:18.482610    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:18.636315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:18.636283    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3130be1-ae0e-4efe-87b7-cddcf3e564c4" path="/var/lib/kubelet/pods/a3130be1-ae0e-4efe-87b7-cddcf3e564c4/volumes"
Nov 03 08:11:19.494095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:19.494026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:19.645772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:19.645737    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:19.789903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:19.779450    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:20.572227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.572194    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:20.825168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.825094    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:20.867320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.867290    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"]
Nov 03 08:11:20.868098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.868078    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a3130be1-ae0e-4efe-87b7-cddcf3e564c4" containerName="test-curl-6985-2630f8"
Nov 03 08:11:20.868208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.868098    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3130be1-ae0e-4efe-87b7-cddcf3e564c4" containerName="test-curl-6985-2630f8"
Nov 03 08:11:20.868311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.868296    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3130be1-ae0e-4efe-87b7-cddcf3e564c4" containerName="test-curl-6985-2630f8"
Nov 03 08:11:20.873243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.873223    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:20.879211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.879186    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2a44ba\""
Nov 03 08:11:20.879872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.879850    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2a44ba-dockercfg-57vx2\""
Nov 03 08:11:20.881702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.881650    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"]
Nov 03 08:11:20.929718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.929691    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"]
Nov 03 08:11:20.934117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.934093    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:20.945822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.945794    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"]
Nov 03 08:11:20.952656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.952624    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:20.960935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.960909    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58t2\" (UniqueName: \"kubernetes.io/projected/7bd31378-736f-4527-a3b9-1fbedbd1dff7-kube-api-access-q58t2\") pod \"payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:20.961109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.960973    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66bh\" (UniqueName: \"kubernetes.io/projected/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-kube-api-access-f66bh\") pod \"payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") " pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:20.961167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.961141    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") " pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:20.961238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:20.961221    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd31378-736f-4527-a3b9-1fbedbd1dff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:21.109144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.081456    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t2\" (UniqueName: \"kubernetes.io/projected/7bd31378-736f-4527-a3b9-1fbedbd1dff7-kube-api-access-q58t2\") pod \"payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:21.109144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.081548    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66bh\" (UniqueName: \"kubernetes.io/projected/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-kube-api-access-f66bh\") pod \"payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") " pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:21.109144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.081604    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") " pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:21.109144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.081674    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd31378-736f-4527-a3b9-1fbedbd1dff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:21.109144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.082864    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd31378-736f-4527-a3b9-1fbedbd1dff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:21.130237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.130201    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") " pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:21.174096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.167702    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58t2\" (UniqueName: \"kubernetes.io/projected/7bd31378-736f-4527-a3b9-1fbedbd1dff7-kube-api-access-q58t2\") pod \"payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:21.174341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.171760    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66bh\" (UniqueName: \"kubernetes.io/projected/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-kube-api-access-f66bh\") pod \"payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") " pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:21.202597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.202565    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:21.257151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.251973    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:21.654657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.654621    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:21.814724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.814692    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"]
Nov 03 08:11:21.816684 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:11:21.816658    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd31378_736f_4527_a3b9_1fbedbd1dff7.slice/crio-038ddd7b12cd77b9cedbcf06cac09618528b94777c4cdd2e7734fa455b023ae7 WatchSource:0}: Error finding container 038ddd7b12cd77b9cedbcf06cac09618528b94777c4cdd2e7734fa455b023ae7: Status 404 returned error can't find the container with id 038ddd7b12cd77b9cedbcf06cac09618528b94777c4cdd2e7734fa455b023ae7
Nov 03 08:11:21.818224 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:11:21.818185    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fe1fbf_a12d_4e8f_9080_0ee66257825d.slice/crio-fa7234aa8e4c1387d7bd7b6819a4db13117ad468bbbaf757b7a0afb7f450ba7a WatchSource:0}: Error finding container fa7234aa8e4c1387d7bd7b6819a4db13117ad468bbbaf757b7a0afb7f450ba7a: Status 404 returned error can't find the container with id fa7234aa8e4c1387d7bd7b6819a4db13117ad468bbbaf757b7a0afb7f450ba7a
Nov 03 08:11:21.819247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:21.819225    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"]
Nov 03 08:11:22.021830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.021791    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs" event={"ID":"c1fe1fbf-a12d-4e8f-9080-0ee66257825d","Type":"ContainerStarted","Data":"15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126"}
Nov 03 08:11:22.022039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.022023    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:22.022127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.022083    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs" event={"ID":"c1fe1fbf-a12d-4e8f-9080-0ee66257825d","Type":"ContainerStarted","Data":"fa7234aa8e4c1387d7bd7b6819a4db13117ad468bbbaf757b7a0afb7f450ba7a"}
Nov 03 08:11:22.023555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.023519    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6" event={"ID":"7bd31378-736f-4527-a3b9-1fbedbd1dff7","Type":"ContainerStarted","Data":"ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad"}
Nov 03 08:11:22.023672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.023570    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6" event={"ID":"7bd31378-736f-4527-a3b9-1fbedbd1dff7","Type":"ContainerStarted","Data":"038ddd7b12cd77b9cedbcf06cac09618528b94777c4cdd2e7734fa455b023ae7"}
Nov 03 08:11:22.023672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.023602    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:22.040511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.040469    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs" podStartSLOduration=2.040454248 podStartE2EDuration="2.040454248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:22.037747538 +0000 UTC m=+1031.540583054" watchObservedRunningTime="2026-11-03 08:11:22.040454248 +0000 UTC m=+1031.543289764"
Nov 03 08:11:22.053759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.053718    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6" podStartSLOduration=2.053705833 podStartE2EDuration="2.053705833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:22.050474635 +0000 UTC m=+1031.553310161" watchObservedRunningTime="2026-11-03 08:11:22.053705833 +0000 UTC m=+1031.556541378"
Nov 03 08:11:22.647692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:22.647658    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:23.481149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:23.481109    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:24.273496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:24.273464    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:25.208054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:25.208017    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:26.043078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:26.043014    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:26.321707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:26.321619    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:26.741950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:26.741922    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:26.936828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:26.936798    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:27.742768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:27.742737    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:28.377280 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:28.377244    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:11:28.454956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:28.454907    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:29.225009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:29.224981    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:29.950246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:29.950187    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:30.688211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:30.688178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:31.667217 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:31.667178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:31.797124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:31.796365    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:31.939291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:31.939260    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:32.937155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:32.937110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:33.043426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:33.043397    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:11:33.044422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:33.044404    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:11:33.489493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:33.489462    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:33.767355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:33.767267    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:34.591114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:34.591035    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:35.312360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:35.312327    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:36.147021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:36.146984    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:36.841284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:36.841252    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:37.750596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:37.750561    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:38.514805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:38.514768    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:39.287130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:39.287044    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:40.024882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:40.024850    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:40.756826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:40.756783    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:41.475309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:41.475277    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:42.233135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:42.233094    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:43.048107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:43.048050    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:43.795305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:43.795262    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:44.453004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:44.452962    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:45.489998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:45.489945    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:46.353589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:46.353551    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:46.508123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:46.508044    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:46.737507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:46.737481    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:46.990140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:46.986652    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:47.112085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:47.108544    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:47.584083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:47.582287    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:47.707698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:47.707653    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:47.956089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:47.940867    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:48.542537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:48.542488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:49.317528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:49.317489    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:49.674134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:49.674023    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:50.209920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:50.209884    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:50.328780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:50.328739    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:50.636917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:50.636843    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:50.852026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:50.851992    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:51.299637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:51.299603    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:51.438660 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:51.438630    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:51.817350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:51.817320    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:52.509713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.509571    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:52.593693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.593663    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"]
Nov 03 08:11:52.614108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.613882    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:52.631848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.631820    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"]
Nov 03 08:11:52.645699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.645671    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:52.700461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.700397    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e255cbee\""
Nov 03 08:11:52.700823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.700804    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e255cbee-dockercfg-pbxsq\""
Nov 03 08:11:52.824417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.810483    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcgk\" (UniqueName: \"kubernetes.io/projected/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-kube-api-access-pbcgk\") pod \"payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") " pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:52.824417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.810683    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") " pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:52.925086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.919143    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcgk\" (UniqueName: \"kubernetes.io/projected/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-kube-api-access-pbcgk\") pod \"payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") " pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:52.925086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.919324    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") " pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:52.925086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.920784    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") " pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:53.003088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:52.995870    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcgk\" (UniqueName: \"kubernetes.io/projected/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-kube-api-access-pbcgk\") pod \"payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") " pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:53.130624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.130543    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:53.337438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.337403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:53.505255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.503589    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"]
Nov 03 08:11:53.567196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.567165    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:53.618900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.618872    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"]
Nov 03 08:11:53.625288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.625266    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:53.629319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.629287    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-178e68\""
Nov 03 08:11:53.629437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.629383    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-178e68-dockercfg-v2pml\""
Nov 03 08:11:53.631684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.631661    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"]
Nov 03 08:11:53.656105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.655770    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:53.656105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.655883    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmtp\" (UniqueName: \"kubernetes.io/projected/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-kube-api-access-wgmtp\") pod \"payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:53.656302 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.656181    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"]
Nov 03 08:11:53.668014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.667994    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:53.675504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.675476    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"]
Nov 03 08:11:53.699448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.696520    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:53.800133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.792930    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:53.800133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.793085    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:53.800133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.793211    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmtp\" (UniqueName: \"kubernetes.io/projected/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-kube-api-access-wgmtp\") pod \"payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:53.800133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.793289    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvzr\" (UniqueName: \"kubernetes.io/projected/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-kube-api-access-zwvzr\") pod \"payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:53.800133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.794522    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:53.869099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.861764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmtp\" (UniqueName: \"kubernetes.io/projected/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-kube-api-access-wgmtp\") pod \"payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:53.897094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.896171    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:53.897094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.896481    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvzr\" (UniqueName: \"kubernetes.io/projected/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-kube-api-access-zwvzr\") pod \"payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:53.913093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.898192    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:53.960820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:53.959495    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:54.141618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.126496    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvzr\" (UniqueName: \"kubernetes.io/projected/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-kube-api-access-zwvzr\") pod \"payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:54.398174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.395251    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:54.459044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.459002    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7" event={"ID":"6490b36f-7cd0-4dfa-8191-5fb41fb632d9","Type":"ContainerStarted","Data":"503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a"}
Nov 03 08:11:54.460231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.460199    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7" event={"ID":"6490b36f-7cd0-4dfa-8191-5fb41fb632d9","Type":"ContainerStarted","Data":"792d3f7d2ac8f3dd63ee1fd74288dd282d626c3c9ad15e75742734b1ed22fbde"}
Nov 03 08:11:54.460798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.460754    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:11:54.481002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.479760    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7" podStartSLOduration=2.479698861 podStartE2EDuration="2.479698861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:54.473962223 +0000 UTC m=+1063.976797737" watchObservedRunningTime="2026-11-03 08:11:54.479698861 +0000 UTC m=+1063.982534377"
Nov 03 08:11:54.507502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.505261    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:54.765999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.765926    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:54.784492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.784389    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"]
Nov 03 08:11:54.792314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.790562    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"]
Nov 03 08:11:54.953228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:54.953190    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:55.467748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.467711    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc" event={"ID":"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf","Type":"ContainerStarted","Data":"ed0f1e9726013e363a247db729534a96edf62a60fc4ddf7efb75a6bfd4ec1cbb"}
Nov 03 08:11:55.467964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.467817    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc" event={"ID":"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf","Type":"ContainerStarted","Data":"81dc8008d3174eeab59b3482fa6e8cbb69d90a7139976cc1687b1472603f28e1"}
Nov 03 08:11:55.467964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.467852    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:11:55.469536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.469506    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm" event={"ID":"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b","Type":"ContainerStarted","Data":"1656089d8efcf7e4fafc7fb494b26b097d903eaebb7b9ff15062e7515453ef8e"}
Nov 03 08:11:55.469636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.469549    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm" event={"ID":"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b","Type":"ContainerStarted","Data":"fc9973fc3f06a844a5ec907f3f364e6d034515ca4fc47a1865b0cf8cb18233f8"}
Nov 03 08:11:55.469703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.469692    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:11:55.501959 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.501912    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc" podStartSLOduration=2.501897981 podStartE2EDuration="2.501897981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:55.498780432 +0000 UTC m=+1065.001615945" watchObservedRunningTime="2026-11-03 08:11:55.501897981 +0000 UTC m=+1065.004733497"
Nov 03 08:11:55.528635 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.528584    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm" podStartSLOduration=2.528570895 podStartE2EDuration="2.528570895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:55.52760938 +0000 UTC m=+1065.030444891" watchObservedRunningTime="2026-11-03 08:11:55.528570895 +0000 UTC m=+1065.031406444"
Nov 03 08:11:55.827271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:55.827179    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:56.128794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.128714    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:56.183763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.183733    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"]
Nov 03 08:11:56.188752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.188700    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.190006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.189981    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"]
Nov 03 08:11:56.194265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.194248    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:56.198112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.198023    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dfb883\""
Nov 03 08:11:56.198247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.198206    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dfb883-dockercfg-rdfzf\""
Nov 03 08:11:56.201346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.201327    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"]
Nov 03 08:11:56.214700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.214676    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"]
Nov 03 08:11:56.312620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.312581    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:56.399287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.399204    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15457759-ee02-483d-954b-f680b9595074-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") " pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:56.399584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.399564    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ntbb\" (UniqueName: \"kubernetes.io/projected/15457759-ee02-483d-954b-f680b9595074-kube-api-access-6ntbb\") pod \"payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") " pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:56.399807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.399792    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s786t\" (UniqueName: \"kubernetes.io/projected/07cde62a-a524-40a7-84e4-5ad1b532787f-kube-api-access-s786t\") pod \"payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.400010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.399994    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cde62a-a524-40a7-84e4-5ad1b532787f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.508328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.505680    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15457759-ee02-483d-954b-f680b9595074-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") " pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:56.508328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.505731    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ntbb\" (UniqueName: \"kubernetes.io/projected/15457759-ee02-483d-954b-f680b9595074-kube-api-access-6ntbb\") pod \"payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") " pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:56.508328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.505817    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s786t\" (UniqueName: \"kubernetes.io/projected/07cde62a-a524-40a7-84e4-5ad1b532787f-kube-api-access-s786t\") pod \"payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.508328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.505901    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cde62a-a524-40a7-84e4-5ad1b532787f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.508328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.507327    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cde62a-a524-40a7-84e4-5ad1b532787f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.558115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.557652    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15457759-ee02-483d-954b-f680b9595074-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") " pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:56.602174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.597932    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s786t\" (UniqueName: \"kubernetes.io/projected/07cde62a-a524-40a7-84e4-5ad1b532787f-kube-api-access-s786t\") pod \"payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.625118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.617727    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ntbb\" (UniqueName: \"kubernetes.io/projected/15457759-ee02-483d-954b-f680b9595074-kube-api-access-6ntbb\") pod \"payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") " pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:56.809232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.809201    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:56.816604 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:56.816587    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:57.107187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.107156    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"]
Nov 03 08:11:57.108611 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:11:57.108574    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cde62a_a524_40a7_84e4_5ad1b532787f.slice/crio-1f57c1e16db327a1f3d6951c29d02f21bab2afd4201ead955e3b065406516771 WatchSource:0}: Error finding container 1f57c1e16db327a1f3d6951c29d02f21bab2afd4201ead955e3b065406516771: Status 404 returned error can't find the container with id 1f57c1e16db327a1f3d6951c29d02f21bab2afd4201ead955e3b065406516771
Nov 03 08:11:57.114625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.114588    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"]
Nov 03 08:11:57.116476 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:11:57.116435    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15457759_ee02_483d_954b_f680b9595074.slice/crio-1a297a3120a4d80dab45d07d41417127dd2ba3a4eedb05c96b966c28ee38b0d6 WatchSource:0}: Error finding container 1a297a3120a4d80dab45d07d41417127dd2ba3a4eedb05c96b966c28ee38b0d6: Status 404 returned error can't find the container with id 1a297a3120a4d80dab45d07d41417127dd2ba3a4eedb05c96b966c28ee38b0d6
Nov 03 08:11:57.281014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.280967    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:57.610187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.608336    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4" event={"ID":"15457759-ee02-483d-954b-f680b9595074","Type":"ContainerStarted","Data":"f11badad52326d3441219a575ef69c0f4c076698b4de70b3f1ac20378a4a33d4"}
Nov 03 08:11:57.610187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.608411    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4" event={"ID":"15457759-ee02-483d-954b-f680b9595074","Type":"ContainerStarted","Data":"1a297a3120a4d80dab45d07d41417127dd2ba3a4eedb05c96b966c28ee38b0d6"}
Nov 03 08:11:57.610187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.610034    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:11:57.626714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.625313    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j" event={"ID":"07cde62a-a524-40a7-84e4-5ad1b532787f","Type":"ContainerStarted","Data":"6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363"}
Nov 03 08:11:57.626714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.625379    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j" event={"ID":"07cde62a-a524-40a7-84e4-5ad1b532787f","Type":"ContainerStarted","Data":"1f57c1e16db327a1f3d6951c29d02f21bab2afd4201ead955e3b065406516771"}
Nov 03 08:11:57.626714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.626234    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:11:57.659149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.657923    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4" podStartSLOduration=1.6579043119999999 podStartE2EDuration="1.657904312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:57.6264869 +0000 UTC m=+1067.129322414" watchObservedRunningTime="2026-11-03 08:11:57.657904312 +0000 UTC m=+1067.160739826"
Nov 03 08:11:57.813099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:57.789483    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j" podStartSLOduration=1.78946198 podStartE2EDuration="1.78946198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:57.68080974 +0000 UTC m=+1067.183645278" watchObservedRunningTime="2026-11-03 08:11:57.78946198 +0000 UTC m=+1067.292297495"
Nov 03 08:11:58.279561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:58.279517    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:11:58.471270 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:11:58.471225    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:11:59.477550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:11:59.477516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:00.668086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:00.663925    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:01.385340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:01.385313    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:02.154792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:02.154755    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:02.850373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:02.850342    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:03.648405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:03.648373    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:04.433502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:04.433466    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:05.141024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:05.140987    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:05.472886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:05.472852    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:12:05.873080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:05.872979    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:06.523044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:06.523017    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:12:06.530683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:06.530659    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:12:06.666275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:06.666247    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:06.799162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:06.798396    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:07.591775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:07.591740    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:07.740556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:07.740520    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:08.699941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:08.699904    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:09.464582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.464519    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:09.691045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.691012    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:12:09.691250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.691121    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:12:09.758368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.758289    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6982-8bd389"]
Nov 03 08:12:09.763365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.763344    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:09.768793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.768769    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfklg\""
Nov 03 08:12:09.774276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.774250    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6982-8bd389"]
Nov 03 08:12:09.877954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.877919    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdhcw\" (UniqueName: \"kubernetes.io/projected/471865fa-c0f5-42cf-a354-8cc0ad28d6eb-kube-api-access-rdhcw\") pod \"test-curl-6982-8bd389\" (UID: \"471865fa-c0f5-42cf-a354-8cc0ad28d6eb\") " pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:09.979404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.979372    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdhcw\" (UniqueName: \"kubernetes.io/projected/471865fa-c0f5-42cf-a354-8cc0ad28d6eb-kube-api-access-rdhcw\") pod \"test-curl-6982-8bd389\" (UID: \"471865fa-c0f5-42cf-a354-8cc0ad28d6eb\") " pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:09.987986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:09.987957    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdhcw\" (UniqueName: \"kubernetes.io/projected/471865fa-c0f5-42cf-a354-8cc0ad28d6eb-kube-api-access-rdhcw\") pod \"test-curl-6982-8bd389\" (UID: \"471865fa-c0f5-42cf-a354-8cc0ad28d6eb\") " pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:10.090808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:10.090725    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:10.282146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:10.282114    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:10.442141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:10.436940    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6982-8bd389"]
Nov 03 08:12:10.458199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:10.457784    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:10.459588 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:12:10.459561    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod471865fa_c0f5_42cf_a354_8cc0ad28d6eb.slice/crio-b7fa789bb87601a906bdb47cb85a7242f937cc529bc2935c00aea65e55b431bb WatchSource:0}: Error finding container b7fa789bb87601a906bdb47cb85a7242f937cc529bc2935c00aea65e55b431bb: Status 404 returned error can't find the container with id b7fa789bb87601a906bdb47cb85a7242f937cc529bc2935c00aea65e55b431bb
Nov 03 08:12:10.746563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:10.746428    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:10.940088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:10.933910    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6982-8bd389" event={"ID":"471865fa-c0f5-42cf-a354-8cc0ad28d6eb","Type":"ContainerStarted","Data":"b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d"}
Nov 03 08:12:10.940088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:10.934033    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6982-8bd389" event={"ID":"471865fa-c0f5-42cf-a354-8cc0ad28d6eb","Type":"ContainerStarted","Data":"b7fa789bb87601a906bdb47cb85a7242f937cc529bc2935c00aea65e55b431bb"}
Nov 03 08:12:11.052686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:11.052562    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6982-8bd389" podStartSLOduration=2.052542557 podStartE2EDuration="2.052542557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:11.048159361 +0000 UTC m=+1080.550994876" watchObservedRunningTime="2026-11-03 08:12:11.052542557 +0000 UTC m=+1080.555378071"
Nov 03 08:12:11.342043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:11.341952    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:11.719250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:11.719214    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6982-8bd389"]
Nov 03 08:12:11.723007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:11.722980    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6982-8bd389"]
Nov 03 08:12:12.375835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:12.375796    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:12.552872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:12.552835    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:12.948139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:12.948103    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6982-8bd389" podUID="471865fa-c0f5-42cf-a354-8cc0ad28d6eb" containerName="test-curl-6982-8bd389" containerID="cri-o://b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d" gracePeriod=2
Nov 03 08:12:13.663321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:13.663286    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:14.656149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:14.648341    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:15.168739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.168713    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:15.259214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.259138    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471865fa-c0f5-42cf-a354-8cc0ad28d6eb-kube-api-access-rdhcw\" (UniqueName: \"kubernetes.io/projected/471865fa-c0f5-42cf-a354-8cc0ad28d6eb-kube-api-access-rdhcw\") pod \"471865fa-c0f5-42cf-a354-8cc0ad28d6eb\" (UID: \"471865fa-c0f5-42cf-a354-8cc0ad28d6eb\") "
Nov 03 08:12:15.261405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.261378    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471865fa-c0f5-42cf-a354-8cc0ad28d6eb-kube-api-access-rdhcw" pod "471865fa-c0f5-42cf-a354-8cc0ad28d6eb" (UID: "471865fa-c0f5-42cf-a354-8cc0ad28d6eb"). InnerVolumeSpecName "kube-api-access-rdhcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:15.360917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.360887    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdhcw\" (UniqueName: \"kubernetes.io/projected/471865fa-c0f5-42cf-a354-8cc0ad28d6eb-kube-api-access-rdhcw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:15.374671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.374645    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:15.978333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.978298    6430 generic.go:356] "Generic (PLEG): container finished" podID="471865fa-c0f5-42cf-a354-8cc0ad28d6eb" containerID="b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d" exitCode=137
Nov 03 08:12:15.978532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.978344    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:15.978532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.978409    6430 scope.go:122] "RemoveContainer" containerID="b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d"
Nov 03 08:12:15.988302 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:12:15.988274    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6982-8bd389\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="471865fa-c0f5-42cf-a354-8cc0ad28d6eb" pod="opendatahub/test-curl-6982-8bd389"
Nov 03 08:12:15.989371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.989356    6430 scope.go:122] "RemoveContainer" containerID="b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d"
Nov 03 08:12:15.989643 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:12:15.989628    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d\": container with ID starting with b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d not found: ID does not exist" containerID="b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d"
Nov 03 08:12:15.989682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:15.989650    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d"} err="failed to get container status \"b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d\": rpc error: code = NotFound desc = could not find container \"b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d\": container with ID starting with b6951c350aa06ce4c3c5a5a9c76ccda069764ffc1c0a246200676e1d9dab8b2d not found: ID does not exist"
Nov 03 08:12:16.246382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:16.246299    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:16.620038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:16.619969    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471865fa-c0f5-42cf-a354-8cc0ad28d6eb" path="/var/lib/kubelet/pods/471865fa-c0f5-42cf-a354-8cc0ad28d6eb/volumes"
Nov 03 08:12:16.946076 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:16.946033    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:17.671939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:17.671907    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:18.408573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:18.408531    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:19.150230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:19.150196    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:19.924008 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:19.923974    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:20.643458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:20.643422    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:21.460742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:21.460705    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:22.342086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:22.338151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:22.546678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:22.546643    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"]
Nov 03 08:12:22.574164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:22.574132    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"]
Nov 03 08:12:22.731806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:22.731770    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:22.994147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:22.990658    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:23.551107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:23.543359    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:24.664213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:24.664176    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:25.974099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:25.965589    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:26.490601 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:26.490557    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:26.733728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:26.733661    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:27.086356 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.086287    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:27.290771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.290733    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:27.482715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.482682    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"]
Nov 03 08:12:27.483781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.483742    6430 cpu_manager.go:414] "containerMap: removing container" podUID="471865fa-c0f5-42cf-a354-8cc0ad28d6eb" containerName="test-curl-6982-8bd389"
Nov 03 08:12:27.483781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.483765    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471865fa-c0f5-42cf-a354-8cc0ad28d6eb" containerName="test-curl-6982-8bd389"
Nov 03 08:12:27.483979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.483964    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="471865fa-c0f5-42cf-a354-8cc0ad28d6eb" containerName="test-curl-6982-8bd389"
Nov 03 08:12:27.489052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.489025    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.510056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.510034    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ed5d1a-openshift-default-dockercfg-5prq6\""
Nov 03 08:12:27.513944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.513916    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:27.522673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.522651    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"]
Nov 03 08:12:27.578090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.573427    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" podUID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" containerName="payload-processing" containerID="cri-o://2e82777dec3e1b7393f21f2642559efd2ef988753d210d18ed3f561d4f578047" gracePeriod=25
Nov 03 08:12:27.640628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.598815    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" podUID="4c997a69-3894-482c-9157-245f4fc76a7f" containerName="payload-pre-processing" containerID="cri-o://5dee8730e2809cdfb41bd08adf2acd07470c9375563182bf7912d4df2a8d8364" gracePeriod=25
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.662592    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-socket\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.662912    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-envoy\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.663136    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-token\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.663345    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-podinfo\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.663808    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352fb60-2c9e-4777-8d2a-22aa012189af-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.663896    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-credential-socket\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.675623    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgv8\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-kube-api-access-4vgv8\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.675802    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-data\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.701103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.680254    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-certs\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.842142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.840076    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:27.852941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.852522    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-podinfo\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.852941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.852706    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352fb60-2c9e-4777-8d2a-22aa012189af-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.852941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.852819    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-credential-socket\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.852941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.852906    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgv8\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-kube-api-access-4vgv8\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.852941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.852943    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-data\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.853368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.853101    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-certs\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.853368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.853220    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-socket\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.853368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.853246    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-envoy\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.853368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.853292    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-token\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.859782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.854498    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-socket\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.874097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.864459    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-certs\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.911414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.875101    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352fb60-2c9e-4777-8d2a-22aa012189af-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.911909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.875350    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-credential-socket\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.912045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.907031    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-podinfo\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.920086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.919878    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-data\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.933041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.932841    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgv8\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-kube-api-access-4vgv8\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.946423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.934041    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-envoy\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:27.946423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:27.942585    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-token\") pod \"e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") " pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:28.113462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:28.113363    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:28.161231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:28.157599    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:28.399181 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:12:28.399099    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:12:28.522951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:28.522925    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"]
Nov 03 08:12:28.525107 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:12:28.525076    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7352fb60_2c9e_4777_8d2a_22aa012189af.slice/crio-4a2309589261e30dbf874753042b4e8e0baa66b6fe61cae63ce53e7fcdf4d070 WatchSource:0}: Error finding container 4a2309589261e30dbf874753042b4e8e0baa66b6fe61cae63ce53e7fcdf4d070: Status 404 returned error can't find the container with id 4a2309589261e30dbf874753042b4e8e0baa66b6fe61cae63ce53e7fcdf4d070
Nov 03 08:12:28.528105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:28.528035    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:12:28.528224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:28.528139    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:12:28.528224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:28.528184    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:12:28.922305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:28.922182    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:29.325443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:29.325409    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" event={"ID":"7352fb60-2c9e-4777-8d2a-22aa012189af","Type":"ContainerStarted","Data":"300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a"}
Nov 03 08:12:29.325981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:29.325477    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" event={"ID":"7352fb60-2c9e-4777-8d2a-22aa012189af","Type":"ContainerStarted","Data":"4a2309589261e30dbf874753042b4e8e0baa66b6fe61cae63ce53e7fcdf4d070"}
Nov 03 08:12:29.354997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:29.354944    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podStartSLOduration=2.35493024 podStartE2EDuration="2.35493024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:29.353577977 +0000 UTC m=+1098.856413495" watchObservedRunningTime="2026-11-03 08:12:29.35493024 +0000 UTC m=+1098.857765754"
Nov 03 08:12:29.914087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:29.905377    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:30.119091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:30.118204    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:30.132086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:30.126425    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:30.333027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:30.332998    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:30.333942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:30.333924    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:12:31.149542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:31.149511    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:31.256820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:31.255975    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" podUID="4c997a69-3894-482c-9157-245f4fc76a7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:12:31.635915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:31.635875    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:31.802977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:31.802937    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:32.304042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:32.303986    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" podUID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:12:32.704218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:32.704191    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:33.128556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.128457    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:33.146456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.146424    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"]
Nov 03 08:12:33.151448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.151429    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.183894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.183865    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad3bdc8-1b67-411b-8842-903becf9baf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.184226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.184155    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ln2\" (UniqueName: \"kubernetes.io/projected/fad3bdc8-1b67-411b-8842-903becf9baf4-kube-api-access-m5ln2\") pod \"payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.192157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.192133    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"]
Nov 03 08:12:33.218816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.218785    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"]
Nov 03 08:12:33.225490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.224870    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:33.227531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.227505    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"]
Nov 03 08:12:33.236432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.236409    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ed5d1a-dockercfg-zsdbb\""
Nov 03 08:12:33.252394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.252360    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ed5d1a\""
Nov 03 08:12:33.392135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.387294    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad3bdc8-1b67-411b-8842-903becf9baf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.392135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.387604    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ln2\" (UniqueName: \"kubernetes.io/projected/fad3bdc8-1b67-411b-8842-903becf9baf4-kube-api-access-m5ln2\") pod \"payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.392135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.389798    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad3bdc8-1b67-411b-8842-903becf9baf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.508550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.508504    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjv8z\" (UniqueName: \"kubernetes.io/projected/184484fb-ece1-4e09-91fd-d6261a76b6eb-kube-api-access-rjv8z\") pod \"payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:33.557159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.557047    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184484fb-ece1-4e09-91fd-d6261a76b6eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:33.557742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.524798    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:33.583311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.577000    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ln2\" (UniqueName: \"kubernetes.io/projected/fad3bdc8-1b67-411b-8842-903becf9baf4-kube-api-access-m5ln2\") pod \"payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.679811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.679715    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184484fb-ece1-4e09-91fd-d6261a76b6eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:33.708605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.708456    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184484fb-ece1-4e09-91fd-d6261a76b6eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:33.710576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.709492    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjv8z\" (UniqueName: \"kubernetes.io/projected/184484fb-ece1-4e09-91fd-d6261a76b6eb-kube-api-access-rjv8z\") pod \"payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:33.750116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.748550    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjv8z\" (UniqueName: \"kubernetes.io/projected/184484fb-ece1-4e09-91fd-d6261a76b6eb-kube-api-access-rjv8z\") pod \"payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:33.766600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.766234    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:33.845456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:33.845406    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:34.601003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:34.600288    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:12:34.627707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:34.627657    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:34.902329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:34.900526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:34.918954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:34.918930    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"]
Nov 03 08:12:34.919573 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:12:34.919534    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184484fb_ece1_4e09_91fd_d6261a76b6eb.slice/crio-bf66a1d9451a074284b06725b51a844f79612e9ed0c881e946144d8ebf2b5f2b WatchSource:0}: Error finding container bf66a1d9451a074284b06725b51a844f79612e9ed0c881e946144d8ebf2b5f2b: Status 404 returned error can't find the container with id bf66a1d9451a074284b06725b51a844f79612e9ed0c881e946144d8ebf2b5f2b
Nov 03 08:12:34.951783 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:34.946426    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"]
Nov 03 08:12:34.955786 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:12:34.954598    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad3bdc8_1b67_411b_8842_903becf9baf4.slice/crio-1fcd895171f2e22caadecd7e6bfae43d4ffa4e24d833aa7d28d1099f7a47b6a3 WatchSource:0}: Error finding container 1fcd895171f2e22caadecd7e6bfae43d4ffa4e24d833aa7d28d1099f7a47b6a3: Status 404 returned error can't find the container with id 1fcd895171f2e22caadecd7e6bfae43d4ffa4e24d833aa7d28d1099f7a47b6a3
Nov 03 08:12:35.735428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.735389    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj" event={"ID":"184484fb-ece1-4e09-91fd-d6261a76b6eb","Type":"ContainerStarted","Data":"435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b"}
Nov 03 08:12:35.735629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.735455    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:35.735629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.735471    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj" event={"ID":"184484fb-ece1-4e09-91fd-d6261a76b6eb","Type":"ContainerStarted","Data":"bf66a1d9451a074284b06725b51a844f79612e9ed0c881e946144d8ebf2b5f2b"}
Nov 03 08:12:35.737161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.737136    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz" event={"ID":"fad3bdc8-1b67-411b-8842-903becf9baf4","Type":"ContainerStarted","Data":"ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840"}
Nov 03 08:12:35.737278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.737176    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz" event={"ID":"fad3bdc8-1b67-411b-8842-903becf9baf4","Type":"ContainerStarted","Data":"1fcd895171f2e22caadecd7e6bfae43d4ffa4e24d833aa7d28d1099f7a47b6a3"}
Nov 03 08:12:35.737278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.737262    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:35.769214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.769167    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj" podStartSLOduration=2.76915491 podStartE2EDuration="2.76915491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:35.765846493 +0000 UTC m=+1105.268682018" watchObservedRunningTime="2026-11-03 08:12:35.76915491 +0000 UTC m=+1105.271990424"
Nov 03 08:12:35.807928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.807885    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz" podStartSLOduration=2.807870855 podStartE2EDuration="2.807870855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:35.806500274 +0000 UTC m=+1105.309335789" watchObservedRunningTime="2026-11-03 08:12:35.807870855 +0000 UTC m=+1105.310706420"
Nov 03 08:12:35.969127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:35.967322    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:37.402087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:37.396779    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:38.757584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:38.757551    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:40.084246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:40.080449    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:40.250826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:40.250794    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:41.047089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:41.040266    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:41.242599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:41.242564    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" podUID="4c997a69-3894-482c-9157-245f4fc76a7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:12:42.010296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:42.010242    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:42.290364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:42.290279    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" podUID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:12:42.829269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:42.829236    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:43.670793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:43.670760    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:44.565966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:44.565938    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:44.729478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:44.729442    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:45.560839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:45.560809    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:46.441181 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:46.441100    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:46.747615 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:46.747523    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:12:46.747951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:46.747933    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:12:47.532749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:47.532711    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:47.910523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:47.910496    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"]
Nov 03 08:12:47.932890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:47.932855    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"]
Nov 03 08:12:47.988577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:47.988302    6430 generic.go:356] "Generic (PLEG): container finished" podID="4c997a69-3894-482c-9157-245f4fc76a7f" containerID="5dee8730e2809cdfb41bd08adf2acd07470c9375563182bf7912d4df2a8d8364" exitCode=0
Nov 03 08:12:47.988577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:47.988523    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" event={"ID":"4c997a69-3894-482c-9157-245f4fc76a7f","Type":"ContainerDied","Data":"5dee8730e2809cdfb41bd08adf2acd07470c9375563182bf7912d4df2a8d8364"}
Nov 03 08:12:47.990876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:47.990855    6430 generic.go:356] "Generic (PLEG): container finished" podID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" containerID="2e82777dec3e1b7393f21f2642559efd2ef988753d210d18ed3f561d4f578047" exitCode=0
Nov 03 08:12:47.991046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:47.991026    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" event={"ID":"5ef0b8fb-90fb-4b71-995f-4f3535c78445","Type":"ContainerDied","Data":"2e82777dec3e1b7393f21f2642559efd2ef988753d210d18ed3f561d4f578047"}
Nov 03 08:12:48.040416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.039786    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:48.130277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.130254    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:12:48.134164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.134143    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:12:48.189923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.189878    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:48.310144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.309367    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c997a69-3894-482c-9157-245f4fc76a7f-kube-api-access-vztz4\" (UniqueName: \"kubernetes.io/projected/4c997a69-3894-482c-9157-245f4fc76a7f-kube-api-access-vztz4\") pod \"4c997a69-3894-482c-9157-245f4fc76a7f\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") "
Nov 03 08:12:48.310144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.309474    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef0b8fb-90fb-4b71-995f-4f3535c78445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef0b8fb-90fb-4b71-995f-4f3535c78445-plugins-config-volume\") pod \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") "
Nov 03 08:12:48.310144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.309546    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c997a69-3894-482c-9157-245f4fc76a7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c997a69-3894-482c-9157-245f4fc76a7f-plugins-config-volume\") pod \"4c997a69-3894-482c-9157-245f4fc76a7f\" (UID: \"4c997a69-3894-482c-9157-245f4fc76a7f\") "
Nov 03 08:12:48.310144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.309631    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef0b8fb-90fb-4b71-995f-4f3535c78445-kube-api-access-xtcfm\" (UniqueName: \"kubernetes.io/projected/5ef0b8fb-90fb-4b71-995f-4f3535c78445-kube-api-access-xtcfm\") pod \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\" (UID: \"5ef0b8fb-90fb-4b71-995f-4f3535c78445\") "
Nov 03 08:12:48.365108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.358337    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef0b8fb-90fb-4b71-995f-4f3535c78445-plugins-config-volume" pod "5ef0b8fb-90fb-4b71-995f-4f3535c78445" (UID: "5ef0b8fb-90fb-4b71-995f-4f3535c78445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:48.365108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.358560    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c997a69-3894-482c-9157-245f4fc76a7f-plugins-config-volume" pod "4c997a69-3894-482c-9157-245f4fc76a7f" (UID: "4c997a69-3894-482c-9157-245f4fc76a7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:48.404172 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.393567    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c997a69-3894-482c-9157-245f4fc76a7f-kube-api-access-vztz4" pod "4c997a69-3894-482c-9157-245f4fc76a7f" (UID: "4c997a69-3894-482c-9157-245f4fc76a7f"). InnerVolumeSpecName "kube-api-access-vztz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:48.404172 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.400297    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef0b8fb-90fb-4b71-995f-4f3535c78445-kube-api-access-xtcfm" pod "5ef0b8fb-90fb-4b71-995f-4f3535c78445" (UID: "5ef0b8fb-90fb-4b71-995f-4f3535c78445"). InnerVolumeSpecName "kube-api-access-xtcfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:48.419086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.418868    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vztz4\" (UniqueName: \"kubernetes.io/projected/4c997a69-3894-482c-9157-245f4fc76a7f-kube-api-access-vztz4\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:48.419086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.418903    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef0b8fb-90fb-4b71-995f-4f3535c78445-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:48.419086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.418931    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c997a69-3894-482c-9157-245f4fc76a7f-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:48.419086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.418945    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtcfm\" (UniqueName: \"kubernetes.io/projected/5ef0b8fb-90fb-4b71-995f-4f3535c78445-kube-api-access-xtcfm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:48.773089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:48.770773    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:49.000886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.000856    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"
Nov 03 08:12:49.001049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.000898    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m" event={"ID":"4c997a69-3894-482c-9157-245f4fc76a7f","Type":"ContainerDied","Data":"0ea7215ae6419e8a080c4f0d4035d973744f9693f02d856ba23d7dade55bc0e5"}
Nov 03 08:12:49.001049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.000970    6430 scope.go:122] "RemoveContainer" containerID="5dee8730e2809cdfb41bd08adf2acd07470c9375563182bf7912d4df2a8d8364"
Nov 03 08:12:49.002470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.002450    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"
Nov 03 08:12:49.002693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.002477    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx" event={"ID":"5ef0b8fb-90fb-4b71-995f-4f3535c78445","Type":"ContainerDied","Data":"0652e0aadb31bf6ce6b82d599e34116820b249ed8100e8c76f9d809b2a56a224"}
Nov 03 08:12:49.012166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.012146    6430 scope.go:122] "RemoveContainer" containerID="2e82777dec3e1b7393f21f2642559efd2ef988753d210d18ed3f561d4f578047"
Nov 03 08:12:49.027774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.027749    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"]
Nov 03 08:12:49.031509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.031483    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-mf6rx"]
Nov 03 08:12:49.050370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.050351    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"]
Nov 03 08:12:49.055904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.055886    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-fch6m"]
Nov 03 08:12:49.637090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:49.625587    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:50.552084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:50.551103    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:50.644551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:50.644508    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c997a69-3894-482c-9157-245f4fc76a7f" path="/var/lib/kubelet/pods/4c997a69-3894-482c-9157-245f4fc76a7f/volumes"
Nov 03 08:12:50.645585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:50.645261    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" path="/var/lib/kubelet/pods/5ef0b8fb-90fb-4b71-995f-4f3535c78445/volumes"
Nov 03 08:12:51.406531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.406496    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"]
Nov 03 08:12:51.407700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.407672    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4c997a69-3894-482c-9157-245f4fc76a7f" containerName="payload-pre-processing"
Nov 03 08:12:51.407700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.407699    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c997a69-3894-482c-9157-245f4fc76a7f" containerName="payload-pre-processing"
Nov 03 08:12:51.407917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.407722    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" containerName="payload-processing"
Nov 03 08:12:51.407917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.407729    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" containerName="payload-processing"
Nov 03 08:12:51.408030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.407972    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c997a69-3894-482c-9157-245f4fc76a7f" containerName="payload-pre-processing"
Nov 03 08:12:51.408030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.407987    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef0b8fb-90fb-4b71-995f-4f3535c78445" containerName="payload-processing"
Nov 03 08:12:51.413570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.413549    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.425143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.425122    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ed5d1a-serving-cert\""
Nov 03 08:12:51.446540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.446512    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"]
Nov 03 08:12:51.470546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.470516    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.470728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.470588    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.470786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.470757    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e014ac-f28f-4861-a732-c5321d301811-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.471010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.470973    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6nl\" (UniqueName: \"kubernetes.io/projected/b9e014ac-f28f-4861-a732-c5321d301811-kube-api-access-pp6nl\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.572322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.572291    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:51.572692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.572573    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e014ac-f28f-4861-a732-c5321d301811-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.572766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.572746    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6nl\" (UniqueName: \"kubernetes.io/projected/b9e014ac-f28f-4861-a732-c5321d301811-kube-api-access-pp6nl\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.572988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.572965    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.573112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.573036    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.573329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.573299    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e014ac-f28f-4861-a732-c5321d301811-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.575346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.575326    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.575456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.575448    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.583733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.583712    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6nl\" (UniqueName: \"kubernetes.io/projected/b9e014ac-f28f-4861-a732-c5321d301811-kube-api-access-pp6nl\") pod \"maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.735613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.735243    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:51.739678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:51.739597    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:52.289342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:52.289288    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"]
Nov 03 08:12:52.594144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:52.594031    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:52.911697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:52.911612    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs" podUID="c1fe1fbf-a12d-4e8f-9080-0ee66257825d" containerName="payload-processing" containerID="cri-o://15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126" gracePeriod=25
Nov 03 08:12:52.933486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:52.933323    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6" podUID="7bd31378-736f-4527-a3b9-1fbedbd1dff7" containerName="payload-pre-processing" containerID="cri-o://ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad" gracePeriod=25
Nov 03 08:12:53.031358 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:12:53.031331    6430 logging.go:55] [core] [Channel #711 SubChannel #712]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.103:50052", ServerName: "10.133.0.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.103:50052: connect: connection refused"
Nov 03 08:12:53.031501 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:12:53.031331    6430 logging.go:55] [core] [Channel #710 SubChannel #713]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.104:50052", ServerName: "10.133.0.104:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.104:50052: connect: connection refused"
Nov 03 08:12:53.246160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.246123    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:12:53.272196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.272165    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:12:53.278135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.278099    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7" event={"ID":"b9e014ac-f28f-4861-a732-c5321d301811","Type":"ContainerStarted","Data":"221eb41a428412ad57e952a269061141c2f798f501cb5b590ddc4eee88dbe9fe"}
Nov 03 08:12:53.278303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.278161    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7" event={"ID":"b9e014ac-f28f-4861-a732-c5321d301811","Type":"ContainerStarted","Data":"3e2bed458bd60f02cf7927c382b97f1a803351bbe3e3c0c46e2bfe63cdc33483"}
Nov 03 08:12:53.278303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.278189    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:53.279812 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.279782    6430 generic.go:356] "Generic (PLEG): container finished" podID="c1fe1fbf-a12d-4e8f-9080-0ee66257825d" containerID="15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126" exitCode=0
Nov 03 08:12:53.279957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.279866    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs" event={"ID":"c1fe1fbf-a12d-4e8f-9080-0ee66257825d","Type":"ContainerDied","Data":"15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126"}
Nov 03 08:12:53.279957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.279937    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs" event={"ID":"c1fe1fbf-a12d-4e8f-9080-0ee66257825d","Type":"ContainerDied","Data":"fa7234aa8e4c1387d7bd7b6819a4db13117ad468bbbaf757b7a0afb7f450ba7a"}
Nov 03 08:12:53.280102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.279970    6430 scope.go:122] "RemoveContainer" containerID="15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126"
Nov 03 08:12:53.280174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.280158    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"
Nov 03 08:12:53.282803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.282769    6430 generic.go:356] "Generic (PLEG): container finished" podID="7bd31378-736f-4527-a3b9-1fbedbd1dff7" containerID="ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad" exitCode=0
Nov 03 08:12:53.282946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.282831    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6" event={"ID":"7bd31378-736f-4527-a3b9-1fbedbd1dff7","Type":"ContainerDied","Data":"ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad"}
Nov 03 08:12:53.282946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.282890    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6" event={"ID":"7bd31378-736f-4527-a3b9-1fbedbd1dff7","Type":"ContainerDied","Data":"038ddd7b12cd77b9cedbcf06cac09618528b94777c4cdd2e7734fa455b023ae7"}
Nov 03 08:12:53.283121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.283103    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"
Nov 03 08:12:53.293794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.293776    6430 scope.go:122] "RemoveContainer" containerID="15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126"
Nov 03 08:12:53.294183 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:12:53.294161    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126\": container with ID starting with 15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126 not found: ID does not exist" containerID="15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126"
Nov 03 08:12:53.294299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.294195    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126"} err="failed to get container status \"15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126\": rpc error: code = NotFound desc = could not find container \"15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126\": container with ID starting with 15b302f53c5eb3ad7cf587f73eb03ad11b1576c384cfbc6a231fb7e580541126 not found: ID does not exist"
Nov 03 08:12:53.294299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.294221    6430 scope.go:122] "RemoveContainer" containerID="ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad"
Nov 03 08:12:53.310953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.310923    6430 scope.go:122] "RemoveContainer" containerID="ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad"
Nov 03 08:12:53.311409 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:12:53.311384    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad\": container with ID starting with ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad not found: ID does not exist" containerID="ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad"
Nov 03 08:12:53.311480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.311422    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad"} err="failed to get container status \"ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad\": rpc error: code = NotFound desc = could not find container \"ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad\": container with ID starting with ede6b16af5cc0f41cc53d0acb459521fc8ec99a51ec700f735d15e530e1a91ad not found: ID does not exist"
Nov 03 08:12:53.322210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.322157    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7" podStartSLOduration=1.773141461 podStartE2EDuration="2.322125571s" totalImagesPullingTime="548.98411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:53.317800552 +0000 UTC m=+1122.820636067" watchObservedRunningTime="2026-11-03 08:12:53.322125571 +0000 UTC m=+1122.824961087"
Nov 03 08:12:53.394308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.394276    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-kube-api-access-f66bh\" (UniqueName: \"kubernetes.io/projected/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-kube-api-access-f66bh\") pod \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") "
Nov 03 08:12:53.394508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.394377    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-plugins-config-volume\") pod \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\" (UID: \"c1fe1fbf-a12d-4e8f-9080-0ee66257825d\") "
Nov 03 08:12:53.394584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.394518    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd31378-736f-4527-a3b9-1fbedbd1dff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd31378-736f-4527-a3b9-1fbedbd1dff7-plugins-config-volume\") pod \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") "
Nov 03 08:12:53.394789 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.394754    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd31378-736f-4527-a3b9-1fbedbd1dff7-kube-api-access-q58t2\" (UniqueName: \"kubernetes.io/projected/7bd31378-736f-4527-a3b9-1fbedbd1dff7-kube-api-access-q58t2\") pod \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\" (UID: \"7bd31378-736f-4527-a3b9-1fbedbd1dff7\") "
Nov 03 08:12:53.394913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.394800    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-plugins-config-volume" pod "c1fe1fbf-a12d-4e8f-9080-0ee66257825d" (UID: "c1fe1fbf-a12d-4e8f-9080-0ee66257825d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:53.394913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.394887    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd31378-736f-4527-a3b9-1fbedbd1dff7-plugins-config-volume" pod "7bd31378-736f-4527-a3b9-1fbedbd1dff7" (UID: "7bd31378-736f-4527-a3b9-1fbedbd1dff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:53.395578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.395477    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:53.395799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.395773    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd31378-736f-4527-a3b9-1fbedbd1dff7-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:53.395900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.395804    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:53.397091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.397049    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd31378-736f-4527-a3b9-1fbedbd1dff7-kube-api-access-q58t2" pod "7bd31378-736f-4527-a3b9-1fbedbd1dff7" (UID: "7bd31378-736f-4527-a3b9-1fbedbd1dff7"). InnerVolumeSpecName "kube-api-access-q58t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:53.397173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.397094    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-kube-api-access-f66bh" pod "c1fe1fbf-a12d-4e8f-9080-0ee66257825d" (UID: "c1fe1fbf-a12d-4e8f-9080-0ee66257825d"). InnerVolumeSpecName "kube-api-access-f66bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:53.528127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.523767    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q58t2\" (UniqueName: \"kubernetes.io/projected/7bd31378-736f-4527-a3b9-1fbedbd1dff7-kube-api-access-q58t2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:53.528127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.523807    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66bh\" (UniqueName: \"kubernetes.io/projected/c1fe1fbf-a12d-4e8f-9080-0ee66257825d-kube-api-access-f66bh\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:53.621784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.621758    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"]
Nov 03 08:12:53.634916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.634879    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6"]
Nov 03 08:12:53.646445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.646413    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"]
Nov 03 08:12:53.652052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:53.652026    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs"]
Nov 03 08:12:54.031354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:54.031318    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-2a44ba-86b8dc9f5f-qkqcs" podUID="c1fe1fbf-a12d-4e8f-9080-0ee66257825d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.104:50052\" within 1s: context deadline exceeded"
Nov 03 08:12:54.031547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:54.031320    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2a44ba-64f45c75f5-42nx6" podUID="7bd31378-736f-4527-a3b9-1fbedbd1dff7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.103:50052\" within 1s: context deadline exceeded"
Nov 03 08:12:54.321087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:54.316848    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:54.619869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:54.619798    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd31378-736f-4527-a3b9-1fbedbd1dff7" path="/var/lib/kubelet/pods/7bd31378-736f-4527-a3b9-1fbedbd1dff7/volumes"
Nov 03 08:12:54.620320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:54.620305    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fe1fbf-a12d-4e8f-9080-0ee66257825d" path="/var/lib/kubelet/pods/c1fe1fbf-a12d-4e8f-9080-0ee66257825d/volumes"
Nov 03 08:12:55.080099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:55.080043    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:55.213902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:55.213871    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:55.395845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:55.391148    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:56.167113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:56.167052    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:56.964080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:56.964038    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:57.138087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:57.136503    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:58.321083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:58.315587    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:12:58.428186 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:12:58.428119    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:12:59.403026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:59.402996    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:12:59.767427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:12:59.767394    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:01.218669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:01.218640    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:01.971147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:01.966250    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:03.209719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:03.209671    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:04.173168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:04.173132    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:05.221522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:05.221457    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:06.111658 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:06.111496    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:07.065088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:07.054641    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:08.122621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:08.122578    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:08.829970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:08.829931    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:08.984167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:08.984127    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:09.138312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:09.138233    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:09.346971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:09.346937    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:09.577083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:09.564827    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:10.682537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:10.682422    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:11.848247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:11.848211    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:12.275201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:12.275171    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:12.572332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:12.572229    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:12.896824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:12.896502    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:13.358263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:13.358225    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:13.543363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:13.543328    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:14.008084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.008012    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:14.166291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.147146    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"]
Nov 03 08:13:14.187093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.168504    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c1fe1fbf-a12d-4e8f-9080-0ee66257825d" containerName="payload-processing"
Nov 03 08:13:14.187368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.187337    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fe1fbf-a12d-4e8f-9080-0ee66257825d" containerName="payload-processing"
Nov 03 08:13:14.187789 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.187769    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7bd31378-736f-4527-a3b9-1fbedbd1dff7" containerName="payload-pre-processing"
Nov 03 08:13:14.222156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.222121    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd31378-736f-4527-a3b9-1fbedbd1dff7" containerName="payload-pre-processing"
Nov 03 08:13:14.222963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.222933    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd31378-736f-4527-a3b9-1fbedbd1dff7" containerName="payload-pre-processing"
Nov 03 08:13:14.223157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.223143    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fe1fbf-a12d-4e8f-9080-0ee66257825d" containerName="payload-processing"
Nov 03 08:13:14.298941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.298856    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:14.439383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.434764    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.456085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.447826    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"]
Nov 03 08:13:14.456085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.454474    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-94e3cb-openshift-default-dockercfg-tjdrp\""
Nov 03 08:13:14.534808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.480048    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5pf\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-kube-api-access-gb5pf\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.534976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-socket\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.535022    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d7575-8abc-452e-ba8d-b0965e6f683e-istiod-ca-cert\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.535139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-credential-socket\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.535179    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-token\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.535328    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-podinfo\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.535427    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-certs\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.535473    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-envoy\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.535578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.535520    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-data\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.678546    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-socket\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.678606    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d7575-8abc-452e-ba8d-b0965e6f683e-istiod-ca-cert\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.678729    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-credential-socket\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.678782    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-token\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.679049    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-podinfo\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.679230    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-certs\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.679270    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-envoy\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.679309    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-data\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.679668    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5pf\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-kube-api-access-gb5pf\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.680998    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-socket\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.681533    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d7575-8abc-452e-ba8d-b0965e6f683e-istiod-ca-cert\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.682376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.681722    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-credential-socket\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.685530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.685503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-podinfo\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.686010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.685989    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-certs\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.693083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.689818    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-data\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.701407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.701381    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-envoy\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.729164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.724243    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5pf\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-kube-api-access-gb5pf\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.749316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.732811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-token\") pod \"e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") " pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:14.830137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.792038    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:14.973598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:14.973569    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:15.164142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.155609    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:15.335903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.335872    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"]
Nov 03 08:13:15.338434 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:13:15.338405    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62d7575_8abc_452e_ba8d_b0965e6f683e.slice/crio-e01a418d49fcc6d0785ef7ee817ce8e4839cad2b0ae2f9c627e6dca12a8e1e21 WatchSource:0}: Error finding container e01a418d49fcc6d0785ef7ee817ce8e4839cad2b0ae2f9c627e6dca12a8e1e21: Status 404 returned error can't find the container with id e01a418d49fcc6d0785ef7ee817ce8e4839cad2b0ae2f9c627e6dca12a8e1e21
Nov 03 08:13:15.341712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.341680    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:13:15.341786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.341748    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:13:15.341786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.341776    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:13:15.864618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.864581    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" event={"ID":"d62d7575-8abc-452e-ba8d-b0965e6f683e","Type":"ContainerStarted","Data":"d29585134e5b7b1446dfffd90579c9a48444dad017e89d6f44506c503a69db70"}
Nov 03 08:13:15.864790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.864646    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" event={"ID":"d62d7575-8abc-452e-ba8d-b0965e6f683e","Type":"ContainerStarted","Data":"e01a418d49fcc6d0785ef7ee817ce8e4839cad2b0ae2f9c627e6dca12a8e1e21"}
Nov 03 08:13:15.899163 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.899097    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podStartSLOduration=2.899079307 podStartE2EDuration="2.899079307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:15.896169044 +0000 UTC m=+1145.399004556" watchObservedRunningTime="2026-11-03 08:13:15.899079307 +0000 UTC m=+1145.401914818"
Nov 03 08:13:15.974394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:15.974358    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:16.458978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:16.458665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:16.675712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:16.675671    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:16.896498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:16.896019    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:16.980008 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:16.979982    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:17.309521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:17.309488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:17.658139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:17.648904    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:17.918678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:17.918595    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:17.920709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:17.920670    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:13:18.581549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:18.581519    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:18.788883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:18.788849    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:19.932381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:19.932354    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:20.066939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.066909    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"]
Nov 03 08:13:20.072188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.072167    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:20.082489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.082464    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-94e3cb-dockercfg-55l62\""
Nov 03 08:13:20.086436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.086413    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"]
Nov 03 08:13:20.115930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.115901    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-94e3cb\""
Nov 03 08:13:20.125850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.125822    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:20.190140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.189578    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznsc\" (UniqueName: \"kubernetes.io/projected/e24415d0-ef79-46c2-bb7f-4f44d44fea97-kube-api-access-lznsc\") pod \"payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:20.190140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.189712    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24415d0-ef79-46c2-bb7f-4f44d44fea97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:20.329206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.329162    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznsc\" (UniqueName: \"kubernetes.io/projected/e24415d0-ef79-46c2-bb7f-4f44d44fea97-kube-api-access-lznsc\") pod \"payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:20.329845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.329819    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24415d0-ef79-46c2-bb7f-4f44d44fea97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:20.330863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.330728    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24415d0-ef79-46c2-bb7f-4f44d44fea97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:20.402085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.398201    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznsc\" (UniqueName: \"kubernetes.io/projected/e24415d0-ef79-46c2-bb7f-4f44d44fea97-kube-api-access-lznsc\") pod \"payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:20.469619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.469491    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:20.700085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:20.692352    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:21.052554 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:13:21.052526    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24415d0_ef79_46c2_bb7f_4f44d44fea97.slice/crio-87219626ee67dc32c42032de602e16388c79f8aef1ea3c8d75c8f487b9637ebd WatchSource:0}: Error finding container 87219626ee67dc32c42032de602e16388c79f8aef1ea3c8d75c8f487b9637ebd: Status 404 returned error can't find the container with id 87219626ee67dc32c42032de602e16388c79f8aef1ea3c8d75c8f487b9637ebd
Nov 03 08:13:21.052977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:21.052770    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"]
Nov 03 08:13:21.760456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:21.760413    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:21.975778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:21.975734    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz" event={"ID":"e24415d0-ef79-46c2-bb7f-4f44d44fea97","Type":"ContainerStarted","Data":"fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a"}
Nov 03 08:13:21.975971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:21.975789    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz" event={"ID":"e24415d0-ef79-46c2-bb7f-4f44d44fea97","Type":"ContainerStarted","Data":"87219626ee67dc32c42032de602e16388c79f8aef1ea3c8d75c8f487b9637ebd"}
Nov 03 08:13:21.975971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:21.975895    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:22.004112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:22.004054    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz" podStartSLOduration=2.004040271 podStartE2EDuration="2.004040271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:22.001544743 +0000 UTC m=+1151.504380258" watchObservedRunningTime="2026-11-03 08:13:22.004040271 +0000 UTC m=+1151.506875782"
Nov 03 08:13:22.963249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:22.962960    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:23.159636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:23.159602    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:23.848103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:23.848041    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:24.535451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:24.535417    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:24.652475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:24.652432    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:25.465313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:25.465280    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:26.088786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:26.088750    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:26.295614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:26.295580    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:26.436144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:26.426635    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:26.768190 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:26.768034    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:27.104707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:27.104626    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:27.772166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:27.771796    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:28.376823 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:13:28.376785    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:13:28.907459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:28.898815    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:29.855463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:29.855431    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:30.799641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:30.799606    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:30.943744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:30.943703    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:31.927637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:31.927597    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:32.564462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:32.564410    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:32.915314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:32.915219    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:33.037297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:33.031465    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:13:33.840736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:33.840700    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:34.171899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:34.171796    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:34.400510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:34.400465    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:34.555957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:34.555919    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:34.965371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:34.965308    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:36.060868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:36.060840    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:36.906679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:36.906644    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:37.737475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:37.737437    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:38.562542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:38.562508    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:38.731188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:38.730789    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:38.838005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:38.837919    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:39.644286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:39.644251    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:40.560346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:40.560315    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:41.577930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:41.577878    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:42.885925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:42.885871    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:43.571739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:43.571609    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:43.971423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:43.970589    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:13:43.982954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:43.982915    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:44.830925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:44.830838    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:45.284349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:45.284322    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"]
Nov 03 08:13:45.347602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:45.347544    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"]
Nov 03 08:13:45.463773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:45.463726    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:45.623749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:45.623657    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:45.988786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:45.988752    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:46.783180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:46.783149    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"]
Nov 03 08:13:46.783608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:46.783364    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy" containerID="cri-o://9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb" gracePeriod=30
Nov 03 08:13:46.787699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:46.787675    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"]
Nov 03 08:13:46.787886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:46.787866    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy" containerID="cri-o://c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc" gracePeriod=30
Nov 03 08:13:46.832493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:46.831533    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:47.555676 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:47.555647    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:48.331677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:48.331643    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:49.150789 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:49.150756    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:50.284888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.284838    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4" podUID="15457759-ee02-483d-954b-f680b9595074" containerName="payload-processing" containerID="cri-o://f11badad52326d3441219a575ef69c0f4c076698b4de70b3f1ac20378a4a33d4" gracePeriod=25
Nov 03 08:13:50.313256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.313228    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:50.379111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.378165    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j" podUID="07cde62a-a524-40a7-84e4-5ad1b532787f" containerName="payload-pre-processing" containerID="cri-o://6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363" gracePeriod=25
Nov 03 08:13:50.516685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.516343    6430 generic.go:356] "Generic (PLEG): container finished" podID="15457759-ee02-483d-954b-f680b9595074" containerID="f11badad52326d3441219a575ef69c0f4c076698b4de70b3f1ac20378a4a33d4" exitCode=0
Nov 03 08:13:50.516685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.516571    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4" event={"ID":"15457759-ee02-483d-954b-f680b9595074","Type":"ContainerDied","Data":"f11badad52326d3441219a575ef69c0f4c076698b4de70b3f1ac20378a4a33d4"}
Nov 03 08:13:50.762049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.762017    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:13:50.808798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.808105    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15457759-ee02-483d-954b-f680b9595074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15457759-ee02-483d-954b-f680b9595074-plugins-config-volume\") pod \"15457759-ee02-483d-954b-f680b9595074\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") "
Nov 03 08:13:50.808798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.808338    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15457759-ee02-483d-954b-f680b9595074-kube-api-access-6ntbb\" (UniqueName: \"kubernetes.io/projected/15457759-ee02-483d-954b-f680b9595074-kube-api-access-6ntbb\") pod \"15457759-ee02-483d-954b-f680b9595074\" (UID: \"15457759-ee02-483d-954b-f680b9595074\") "
Nov 03 08:13:50.808798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.808554    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15457759-ee02-483d-954b-f680b9595074-plugins-config-volume" pod "15457759-ee02-483d-954b-f680b9595074" (UID: "15457759-ee02-483d-954b-f680b9595074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:13:50.809492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.809321    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15457759-ee02-483d-954b-f680b9595074-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:50.811028 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.811006    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15457759-ee02-483d-954b-f680b9595074-kube-api-access-6ntbb" pod "15457759-ee02-483d-954b-f680b9595074" (UID: "15457759-ee02-483d-954b-f680b9595074"). InnerVolumeSpecName "kube-api-access-6ntbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:50.858596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.858567    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:13:50.910518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.910480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07cde62a-a524-40a7-84e4-5ad1b532787f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cde62a-a524-40a7-84e4-5ad1b532787f-plugins-config-volume\") pod \"07cde62a-a524-40a7-84e4-5ad1b532787f\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") "
Nov 03 08:13:50.910707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.910559    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07cde62a-a524-40a7-84e4-5ad1b532787f-kube-api-access-s786t\" (UniqueName: \"kubernetes.io/projected/07cde62a-a524-40a7-84e4-5ad1b532787f-kube-api-access-s786t\") pod \"07cde62a-a524-40a7-84e4-5ad1b532787f\" (UID: \"07cde62a-a524-40a7-84e4-5ad1b532787f\") "
Nov 03 08:13:50.911382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.911351    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07cde62a-a524-40a7-84e4-5ad1b532787f-plugins-config-volume" pod "07cde62a-a524-40a7-84e4-5ad1b532787f" (UID: "07cde62a-a524-40a7-84e4-5ad1b532787f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:13:50.911382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.911370    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ntbb\" (UniqueName: \"kubernetes.io/projected/15457759-ee02-483d-954b-f680b9595074-kube-api-access-6ntbb\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:50.913744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:50.913714    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cde62a-a524-40a7-84e4-5ad1b532787f-kube-api-access-s786t" pod "07cde62a-a524-40a7-84e4-5ad1b532787f" (UID: "07cde62a-a524-40a7-84e4-5ad1b532787f"). InnerVolumeSpecName "kube-api-access-s786t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:51.013721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.013675    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cde62a-a524-40a7-84e4-5ad1b532787f-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:51.013721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.013720    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s786t\" (UniqueName: \"kubernetes.io/projected/07cde62a-a524-40a7-84e4-5ad1b532787f-kube-api-access-s786t\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:51.370081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.369749    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:51.527230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.527187    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"
Nov 03 08:13:51.527435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.527223    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4" event={"ID":"15457759-ee02-483d-954b-f680b9595074","Type":"ContainerDied","Data":"1a297a3120a4d80dab45d07d41417127dd2ba3a4eedb05c96b966c28ee38b0d6"}
Nov 03 08:13:51.527435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.527299    6430 scope.go:122] "RemoveContainer" containerID="f11badad52326d3441219a575ef69c0f4c076698b4de70b3f1ac20378a4a33d4"
Nov 03 08:13:51.529167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.528951    6430 generic.go:356] "Generic (PLEG): container finished" podID="07cde62a-a524-40a7-84e4-5ad1b532787f" containerID="6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363" exitCode=0
Nov 03 08:13:51.529702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.529650    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j" event={"ID":"07cde62a-a524-40a7-84e4-5ad1b532787f","Type":"ContainerDied","Data":"6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363"}
Nov 03 08:13:51.529852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.529731    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j" event={"ID":"07cde62a-a524-40a7-84e4-5ad1b532787f","Type":"ContainerDied","Data":"1f57c1e16db327a1f3d6951c29d02f21bab2afd4201ead955e3b065406516771"}
Nov 03 08:13:51.529852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.529810    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"
Nov 03 08:13:51.546568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.546467    6430 scope.go:122] "RemoveContainer" containerID="6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363"
Nov 03 08:13:51.559568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.559535    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"]
Nov 03 08:13:51.562269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.562249    6430 scope.go:122] "RemoveContainer" containerID="6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363"
Nov 03 08:13:51.562630 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:13:51.562604    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363\": container with ID starting with 6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363 not found: ID does not exist" containerID="6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363"
Nov 03 08:13:51.562737 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.562643    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363"} err="failed to get container status \"6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363\": rpc error: code = NotFound desc = could not find container \"6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363\": container with ID starting with 6e4eb82595b413c3684b787820fcc543ffd95321519f70ec7d8db62519834363 not found: ID does not exist"
Nov 03 08:13:51.563677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.563656    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dfb883-54bc569df9-r5kr4"]
Nov 03 08:13:51.570371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.570349    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"]
Nov 03 08:13:51.573985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:51.573926    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dfb883-8bcf7b6c9-7kv6j"]
Nov 03 08:13:52.049442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.049401    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:52.162352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.152478    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.246654    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-kube-api-access-qnmz2\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-kube-api-access-qnmz2\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.246749    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870199b5-d3ba-473a-97e9-da52cfbeeee0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870199b5-d3ba-473a-97e9-da52cfbeeee0-istiod-ca-cert\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.246784    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-podinfo\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.246815    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-data\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.247146    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-token\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-token\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.247235    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-socket\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.247304    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-envoy\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.247392    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-credential-socket\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.252445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.247460    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-certs\") pod \"870199b5-d3ba-473a-97e9-da52cfbeeee0\" (UID: \"870199b5-d3ba-473a-97e9-da52cfbeeee0\") "
Nov 03 08:13:52.254236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.254185    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-certs" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.258618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.258007    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-socket" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.258618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.258186    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-data" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.258962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.258888    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-credential-socket" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.259773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.259399    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870199b5-d3ba-473a-97e9-da52cfbeeee0-istiod-ca-cert" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:13:52.267522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.267491    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-podinfo" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:13:52.267964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.267951    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-envoy" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.268178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.268163    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-kube-api-access-qnmz2" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "kube-api-access-qnmz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:52.269588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.269455    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-token" pod "870199b5-d3ba-473a-97e9-da52cfbeeee0" (UID: "870199b5-d3ba-473a-97e9-da52cfbeeee0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:52.274370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.274351    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:13:52.349612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.349531    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-token\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-token\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.349612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.349596    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-credential-socket\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.349819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.349694    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-kube-api-access-cxmtd\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-kube-api-access-cxmtd\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.349819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.349780    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34142c2c-a6ec-433b-9719-8e7a382fc660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34142c2c-a6ec-433b-9719-8e7a382fc660-istiod-ca-cert\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.349934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.349843    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-podinfo\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.349934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.349878    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-envoy\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.350020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.350003    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-certs\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.350145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.350125    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-data\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.350245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.350184    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-socket\") pod \"34142c2c-a6ec-433b-9719-8e7a382fc660\" (UID: \"34142c2c-a6ec-433b-9719-8e7a382fc660\") "
Nov 03 08:13:52.352488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.352275    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-certs" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.352608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.352489    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-token" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:52.352608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.352536    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-data" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.352608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.352600    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34142c2c-a6ec-433b-9719-8e7a382fc660-istiod-ca-cert" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:13:52.352834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.352800    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-socket" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.352989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.352950    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-credential-socket" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.353403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.350959    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870199b5-d3ba-473a-97e9-da52cfbeeee0-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.353539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.353524    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.353642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.353628    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.353740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.353729    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.353833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.353821    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.353920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.353909    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.354007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.353994    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.354120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.354107    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870199b5-d3ba-473a-97e9-da52cfbeeee0-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.354220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.354207    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnmz2\" (UniqueName: \"kubernetes.io/projected/870199b5-d3ba-473a-97e9-da52cfbeeee0-kube-api-access-qnmz2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.355153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.355027    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-envoy" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:52.355408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.355387    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-podinfo" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:13:52.357292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.357270    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-kube-api-access-cxmtd" pod "34142c2c-a6ec-433b-9719-8e7a382fc660" (UID: "34142c2c-a6ec-433b-9719-8e7a382fc660"). InnerVolumeSpecName "kube-api-access-cxmtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:52.456630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456591    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.456630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456628    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.457053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456643    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxmtd\" (UniqueName: \"kubernetes.io/projected/34142c2c-a6ec-433b-9719-8e7a382fc660-kube-api-access-cxmtd\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.457053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456660    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34142c2c-a6ec-433b-9719-8e7a382fc660-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.457053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456676    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.457053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456697    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.457053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456711    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.457053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456724    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.457053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.456738    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34142c2c-a6ec-433b-9719-8e7a382fc660-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:52.538035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.538001    6430 generic.go:356] "Generic (PLEG): container finished" podID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerID="9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb" exitCode=0
Nov 03 08:13:52.538216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.538082    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"
Nov 03 08:13:52.538216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.538090    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" event={"ID":"870199b5-d3ba-473a-97e9-da52cfbeeee0","Type":"ContainerDied","Data":"9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb"}
Nov 03 08:13:52.538216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.538152    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr" event={"ID":"870199b5-d3ba-473a-97e9-da52cfbeeee0","Type":"ContainerDied","Data":"47334e89ce95a7b49cbf47fbff05e0356af7ae2870d31b84987c6e22a94bd352"}
Nov 03 08:13:52.538216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.538183    6430 scope.go:122] "RemoveContainer" containerID="9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb"
Nov 03 08:13:52.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.541078    6430 generic.go:356] "Generic (PLEG): container finished" podID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerID="c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc" exitCode=0
Nov 03 08:13:52.541252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.541127    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" event={"ID":"34142c2c-a6ec-433b-9719-8e7a382fc660","Type":"ContainerDied","Data":"c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc"}
Nov 03 08:13:52.541252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.541184    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp" event={"ID":"34142c2c-a6ec-433b-9719-8e7a382fc660","Type":"ContainerDied","Data":"ec8eb45a0f2ef13b9735de703609b87f2c8d2a5892f57f16c906f334e06ff641"}
Nov 03 08:13:52.541252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.541133    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"
Nov 03 08:13:52.550628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.550592    6430 scope.go:122] "RemoveContainer" containerID="9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb"
Nov 03 08:13:52.550900 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:13:52.550872    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb\": container with ID starting with 9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb not found: ID does not exist" containerID="9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb"
Nov 03 08:13:52.550990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.550911    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb"} err="failed to get container status \"9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb\": rpc error: code = NotFound desc = could not find container \"9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb\": container with ID starting with 9e173d96142d5ff1ae3082ad955ef790d3bba48e1ca9096f8643e5c29d0defbb not found: ID does not exist"
Nov 03 08:13:52.550990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.550933    6430 scope.go:122] "RemoveContainer" containerID="c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc"
Nov 03 08:13:52.561922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.561908    6430 scope.go:122] "RemoveContainer" containerID="c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc"
Nov 03 08:13:52.562179 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:13:52.562163    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc\": container with ID starting with c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc not found: ID does not exist" containerID="c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc"
Nov 03 08:13:52.562241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.562188    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc"} err="failed to get container status \"c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc\": rpc error: code = NotFound desc = could not find container \"c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc\": container with ID starting with c76fd16c64d3093e1db0a4fcfed392f967bfb0cb4f113f84825f78644e6638cc not found: ID does not exist"
Nov 03 08:13:52.571862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.571832    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"]
Nov 03 08:13:52.575527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.575509    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-bwgzr"]
Nov 03 08:13:52.584751 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.584733    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"]
Nov 03 08:13:52.590620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.590601    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-wdnkp"]
Nov 03 08:13:52.618769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.618700    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07cde62a-a524-40a7-84e4-5ad1b532787f" path="/var/lib/kubelet/pods/07cde62a-a524-40a7-84e4-5ad1b532787f/volumes"
Nov 03 08:13:52.619264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.619245    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15457759-ee02-483d-954b-f680b9595074" path="/var/lib/kubelet/pods/15457759-ee02-483d-954b-f680b9595074/volumes"
Nov 03 08:13:52.619759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.619741    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" path="/var/lib/kubelet/pods/34142c2c-a6ec-433b-9719-8e7a382fc660/volumes"
Nov 03 08:13:52.620559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:52.620540    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" path="/var/lib/kubelet/pods/870199b5-d3ba-473a-97e9-da52cfbeeee0/volumes"
Nov 03 08:13:53.038095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.030344    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:53.554047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.553963    6430 generic.go:356] "Generic (PLEG): container finished" podID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerID="300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a" exitCode=0
Nov 03 08:13:53.554047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.554000    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" event={"ID":"7352fb60-2c9e-4777-8d2a-22aa012189af","Type":"ContainerDied","Data":"300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a"}
Nov 03 08:13:53.554604 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.554480    6430 scope.go:122] "RemoveContainer" containerID="300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a"
Nov 03 08:13:53.556433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.556403    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:13:53.556507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.556494    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:13:53.556548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.556536    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:13:53.852086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:53.851656    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:54.188630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.188599    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:54.323940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.323910    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:54.563524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.563439    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" event={"ID":"7352fb60-2c9e-4777-8d2a-22aa012189af","Type":"ContainerStarted","Data":"23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da"}
Nov 03 08:13:54.648848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.648819    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:54.751148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.750587    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6982-fb3e29"]
Nov 03 08:13:54.758085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.752785    6430 cpu_manager.go:414] "containerMap: removing container" podUID="15457759-ee02-483d-954b-f680b9595074" containerName="payload-processing"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774208    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15457759-ee02-483d-954b-f680b9595074" containerName="payload-processing"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774354    6430 cpu_manager.go:414] "containerMap: removing container" podUID="07cde62a-a524-40a7-84e4-5ad1b532787f" containerName="payload-pre-processing"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774364    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07cde62a-a524-40a7-84e4-5ad1b532787f" containerName="payload-pre-processing"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774422    6430 cpu_manager.go:414] "containerMap: removing container" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774430    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774477    6430 cpu_manager.go:414] "containerMap: removing container" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774501    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774866    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="07cde62a-a524-40a7-84e4-5ad1b532787f" containerName="payload-pre-processing"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774906    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="870199b5-d3ba-473a-97e9-da52cfbeeee0" containerName="istio-proxy"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774921    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="34142c2c-a6ec-433b-9719-8e7a382fc660" containerName="istio-proxy"
Nov 03 08:13:54.789122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.774935    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="15457759-ee02-483d-954b-f680b9595074" containerName="payload-processing"
Nov 03 08:13:54.860644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.860552    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6982-fb3e29"]
Nov 03 08:13:54.860819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.860639    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:13:54.863998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.863967    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfklg\""
Nov 03 08:13:54.940350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:54.940306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz8v4\" (UniqueName: \"kubernetes.io/projected/709d59f0-6c85-46c8-990b-f6db14fcf4cc-kube-api-access-qz8v4\") pod \"test-curl-6982-fb3e29\" (UID: \"709d59f0-6c85-46c8-990b-f6db14fcf4cc\") " pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:13:55.042137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.042099    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz8v4\" (UniqueName: \"kubernetes.io/projected/709d59f0-6c85-46c8-990b-f6db14fcf4cc-kube-api-access-qz8v4\") pod \"test-curl-6982-fb3e29\" (UID: \"709d59f0-6c85-46c8-990b-f6db14fcf4cc\") " pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:13:55.051318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.051283    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz8v4\" (UniqueName: \"kubernetes.io/projected/709d59f0-6c85-46c8-990b-f6db14fcf4cc-kube-api-access-qz8v4\") pod \"test-curl-6982-fb3e29\" (UID: \"709d59f0-6c85-46c8-990b-f6db14fcf4cc\") " pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:13:55.122313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.122208    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 08:13:55.122313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.122280    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 08:13:55.122715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.122691    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:13:55.181764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.181699    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:13:55.511434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.503936    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:55.547087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.543287    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6982-fb3e29"]
Nov 03 08:13:55.684428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:55.684107    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6982-fb3e29" event={"ID":"709d59f0-6c85-46c8-990b-f6db14fcf4cc","Type":"ContainerStarted","Data":"5671a123a5f89960241eaddb71b5459cfcaa1699345820c2d2a84efb649cb7e8"}
Nov 03 08:13:56.119151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:56.115902    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 08:13:56.119151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:56.115967    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 08:13:56.241779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:56.241741    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:56.692764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:56.692725    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6982-fb3e29" event={"ID":"709d59f0-6c85-46c8-990b-f6db14fcf4cc","Type":"ContainerStarted","Data":"35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83"}
Nov 03 08:13:57.117017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:57.114499    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 08:13:57.117017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:57.114564    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 08:13:57.381495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:57.381409    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:57.816854 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:57.815219    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6982-fb3e29" podStartSLOduration=3.815201133 podStartE2EDuration="3.815201133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:56.722264941 +0000 UTC m=+1186.225100458" watchObservedRunningTime="2026-11-03 08:13:57.815201133 +0000 UTC m=+1187.318036648"
Nov 03 08:13:57.829925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:57.829881    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6982-fb3e29"]
Nov 03 08:13:57.836948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:57.836915    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6982-fb3e29"]
Nov 03 08:13:58.129620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:58.129514    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 08:13:58.129620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:58.129579    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 08:13:58.379082 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:13:58.378702    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:13:58.665692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:58.656419    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:58.738102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:58.738043    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6982-fb3e29" podUID="709d59f0-6c85-46c8-990b-f6db14fcf4cc" containerName="test-curl-6982-fb3e29" containerID="cri-o://35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83" gracePeriod=2
Nov 03 08:13:58.893536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:58.893499    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:59.115528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:59.115487    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 08:13:59.115833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:59.115803    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 08:13:59.493093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:59.493050    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:59.639337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:59.639295    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:13:59.966104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:13:59.965922    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:00.120532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:00.120497    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:00.121950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:00.121921    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 08:14:00.122182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:00.122158    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 08:14:00.338387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:00.338291    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:14:00.763047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:00.763010    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:01.120469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.120438    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:14:01.121641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.121619    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:14:01.141820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.141792    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:14:01.197520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.197480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709d59f0-6c85-46c8-990b-f6db14fcf4cc-kube-api-access-qz8v4\" (UniqueName: \"kubernetes.io/projected/709d59f0-6c85-46c8-990b-f6db14fcf4cc-kube-api-access-qz8v4\") pod \"709d59f0-6c85-46c8-990b-f6db14fcf4cc\" (UID: \"709d59f0-6c85-46c8-990b-f6db14fcf4cc\") "
Nov 03 08:14:01.202198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.202168    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709d59f0-6c85-46c8-990b-f6db14fcf4cc-kube-api-access-qz8v4" pod "709d59f0-6c85-46c8-990b-f6db14fcf4cc" (UID: "709d59f0-6c85-46c8-990b-f6db14fcf4cc"). InnerVolumeSpecName "kube-api-access-qz8v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:01.299952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.299866    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz8v4\" (UniqueName: \"kubernetes.io/projected/709d59f0-6c85-46c8-990b-f6db14fcf4cc-kube-api-access-qz8v4\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:01.639142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.638364    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:01.766458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.766420    6430 generic.go:356] "Generic (PLEG): container finished" podID="709d59f0-6c85-46c8-990b-f6db14fcf4cc" containerID="35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83" exitCode=137
Nov 03 08:14:01.766651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.766480    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:14:01.766651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.766526    6430 scope.go:122] "RemoveContainer" containerID="35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83"
Nov 03 08:14:01.780609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.780539    6430 scope.go:122] "RemoveContainer" containerID="35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83"
Nov 03 08:14:01.780908 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:01.780878    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83\": container with ID starting with 35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83 not found: ID does not exist" containerID="35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83"
Nov 03 08:14:01.780984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:01.780921    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83"} err="failed to get container status \"35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83\": rpc error: code = NotFound desc = could not find container \"35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83\": container with ID starting with 35dc2d0b6c29cbbe9b00969e159e6ad94bc6fced814819e8bb14db1ff024fb83 not found: ID does not exist"
Nov 03 08:14:01.782034 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:01.782009    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6982-fb3e29\" is forbidden: User \"system:node:ip-10-0-140-232.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-232.us-east-2.compute.internal' and this object" podUID="709d59f0-6c85-46c8-990b-f6db14fcf4cc" pod="opendatahub/test-curl-6982-fb3e29"
Nov 03 08:14:02.343370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:02.343333    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:02.619033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:02.618954    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709d59f0-6c85-46c8-990b-f6db14fcf4cc" path="/var/lib/kubelet/pods/709d59f0-6c85-46c8-990b-f6db14fcf4cc/volumes"
Nov 03 08:14:03.045980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:03.045953    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:03.772285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:03.772253    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:04.550354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:04.550326    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:05.314115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.314077    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:05.883585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.883531    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"]
Nov 03 08:14:05.884822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.884798    6430 cpu_manager.go:414] "containerMap: removing container" podUID="709d59f0-6c85-46c8-990b-f6db14fcf4cc" containerName="test-curl-6982-fb3e29"
Nov 03 08:14:05.884822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.884821    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709d59f0-6c85-46c8-990b-f6db14fcf4cc" containerName="test-curl-6982-fb3e29"
Nov 03 08:14:05.885024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.884976    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="709d59f0-6c85-46c8-990b-f6db14fcf4cc" containerName="test-curl-6982-fb3e29"
Nov 03 08:14:05.900456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.900424    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"]
Nov 03 08:14:05.900712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.900691    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:05.905027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.905000    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:05.909130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.909106    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"]
Nov 03 08:14:05.909320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.909305    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"]
Nov 03 08:14:05.909501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.909490    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:05.984864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:05.984830    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dfb883-openshift-default-dockercfg-mtfb5\""
Nov 03 08:14:06.018957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.018916    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019235    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psf9d\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-kube-api-access-psf9d\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019340    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.019457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019406    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.019515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019455    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.019515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019495    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019561    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7019add-ed7c-4cc8-bcfc-0492666161d3-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019592    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019618    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019690    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019723    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019821    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.019888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019874    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kcs\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-kube-api-access-25kcs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.019995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019905    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.019995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019943    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.019995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.019988    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.020178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.020080    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.020237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.020226    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.134130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.129047    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.134130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.130177    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.134130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.130268    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.134130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.130717    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.134130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.130641    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.134130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.131149    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.144201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.144168    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7019add-ed7c-4cc8-bcfc-0492666161d3-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.145760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.145736    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.146741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.146326    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.146964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.146945    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.147410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.147392    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.152159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.145634    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7019add-ed7c-4cc8-bcfc-0492666161d3-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.152675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.147331    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.152834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.152548    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.152951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.148049    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.153119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.153102    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kcs\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-kube-api-access-25kcs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.153570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.153552    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.154043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.154025    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.154668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.154518    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.154842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.154824    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.159335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.155208    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.159784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.159761    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.160256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.160236    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.173414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.173379    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.173682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.173667    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psf9d\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-kube-api-access-psf9d\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.174435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.173928    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.174828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.163694    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.174932 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.173845    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.174932 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.174638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.191220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.190013    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.214757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.199835    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kcs\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-kube-api-access-25kcs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.214757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.200312    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.214757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.210790    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.231099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.223474    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.231099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.224003    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.241681 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.241447    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:06.251327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.250732    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psf9d\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-kube-api-access-psf9d\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.542229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.528873    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:06.586659 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.549677    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:06.709955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.709872    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:06.746177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.745816    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"]
Nov 03 08:14:06.750987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.750954    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:06.751122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.751033    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:06.751122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.751085    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:06.923210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.901389    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:06.933709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.932153    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" event={"ID":"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137","Type":"ContainerStarted","Data":"ed69582be904ddd5bba3887d4d311fd11beb10f174aff684154a448495c8cd8e"}
Nov 03 08:14:06.972246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.969585    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"]
Nov 03 08:14:06.975962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.975918    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:06.976107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.976004    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:06.976107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:06.976044    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:07.213087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:07.209624    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:07.546361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:07.546264    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:07.676863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:07.676821    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:07.849567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:07.849476    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:07.985423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:07.985377    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" event={"ID":"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137","Type":"ContainerStarted","Data":"acc970c734879c26f77be2f6f2925204f8bef852e5022be204c12519c582de0b"}
Nov 03 08:14:07.991218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:07.989920    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" event={"ID":"b7019add-ed7c-4cc8-bcfc-0492666161d3","Type":"ContainerStarted","Data":"ace50e7b444d65ae218b150a8bd7b9dedcaf67307a216efd560baeabbef15f6f"}
Nov 03 08:14:07.991218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:07.989982    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" event={"ID":"b7019add-ed7c-4cc8-bcfc-0492666161d3","Type":"ContainerStarted","Data":"fa3664600adbee1e4ce19296316658ad38b3b65564d5194727c2555c68bb89df"}
Nov 03 08:14:08.014736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.009520    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podStartSLOduration=3.009500385 podStartE2EDuration="3.009500385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:08.007249232 +0000 UTC m=+1197.510084763" watchObservedRunningTime="2026-11-03 08:14:08.009500385 +0000 UTC m=+1197.512335900"
Nov 03 08:14:08.245976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.245431    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:08.266243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.265651    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:14:08.266243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.265728    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 08:14:08.366927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.365967    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:08.522138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.520948    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:08.545085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.544883    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:08.545280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.545095    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:14:08.545280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:08.545143    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 08:14:09.083284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.083184    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:09.332833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.332797    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:14:09.359437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.359346    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 08:14:09.361737 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.361706    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:09.544548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.544511    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:14:09.544794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.544767    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 08:14:09.860474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.860399    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podStartSLOduration=4.860382561 podStartE2EDuration="4.860382561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:08.041394496 +0000 UTC m=+1197.544230010" watchObservedRunningTime="2026-11-03 08:14:09.860382561 +0000 UTC m=+1199.363218078"
Nov 03 08:14:09.862294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.862265    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"]
Nov 03 08:14:09.974123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:09.973923    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:10.131363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:10.131249    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:10.249091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:10.244685    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:14:10.249091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:10.244758    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 08:14:10.318123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:10.318085    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:10.539545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:10.539508    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:14:10.539850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:10.539820    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 08:14:11.167091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:11.160431    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:11.244357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:11.244319    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:14:11.244621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:11.244586    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 08:14:11.540084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:11.539754    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:14:11.540084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:11.539837    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 08:14:11.961670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:11.961638    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:12.253145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:12.251207    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:14:12.253145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:12.251279    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 08:14:12.543512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:12.543421    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:14:12.544549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:12.544422    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:12.545139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:12.545115    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 08:14:12.956331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:12.956293    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:13.247233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:13.247139    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:14:13.247470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:13.247442    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 08:14:13.545148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:13.541618    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:14:13.545148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:13.541686    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 08:14:13.861959 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:13.861871    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:14.022088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.021795    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:14.221570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.221530    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"]
Nov 03 08:14:14.221790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.221764    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy" containerID="cri-o://ace50e7b444d65ae218b150a8bd7b9dedcaf67307a216efd560baeabbef15f6f" gracePeriod=30
Nov 03 08:14:14.245487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.245450    6430 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:14:14.245768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.245736    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 08:14:14.431106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.420814    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:14.871108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.867099    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7" podUID="6490b36f-7cd0-4dfa-8191-5fb41fb632d9" containerName="payload-processing" containerID="cri-o://503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a" gracePeriod=25
Nov 03 08:14:14.966750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.966617    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:14:15.010112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.993858    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:15.010112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.998771    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"]
Nov 03 08:14:15.010112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:14.999430    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy" containerID="cri-o://acc970c734879c26f77be2f6f2925204f8bef852e5022be204c12519c582de0b" gracePeriod=30
Nov 03 08:14:15.092768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.089593    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:14:15.139126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.137538    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:15.417770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.417701    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:14:15.484095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.483941    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:15.519223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.519179    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-plugins-config-volume" pod "6490b36f-7cd0-4dfa-8191-5fb41fb632d9" (UID: "6490b36f-7cd0-4dfa-8191-5fb41fb632d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:15.519409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.519260    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-plugins-config-volume\") pod \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") "
Nov 03 08:14:15.525575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.520173    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-kube-api-access-pbcgk\" (UniqueName: \"kubernetes.io/projected/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-kube-api-access-pbcgk\") pod \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\" (UID: \"6490b36f-7cd0-4dfa-8191-5fb41fb632d9\") "
Nov 03 08:14:15.525575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.521952    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:15.531200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.528414    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-kube-api-access-pbcgk" pod "6490b36f-7cd0-4dfa-8191-5fb41fb632d9" (UID: "6490b36f-7cd0-4dfa-8191-5fb41fb632d9"). InnerVolumeSpecName "kube-api-access-pbcgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:15.640088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.623036    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:15.679629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.641605    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbcgk\" (UniqueName: \"kubernetes.io/projected/6490b36f-7cd0-4dfa-8191-5fb41fb632d9-kube-api-access-pbcgk\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:15.689823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.689790    6430 generic.go:356] "Generic (PLEG): container finished" podID="6490b36f-7cd0-4dfa-8191-5fb41fb632d9" containerID="503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a" exitCode=0
Nov 03 08:14:15.690141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.690118    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7" event={"ID":"6490b36f-7cd0-4dfa-8191-5fb41fb632d9","Type":"ContainerDied","Data":"503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a"}
Nov 03 08:14:15.690312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.690296    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7" event={"ID":"6490b36f-7cd0-4dfa-8191-5fb41fb632d9","Type":"ContainerDied","Data":"792d3f7d2ac8f3dd63ee1fd74288dd282d626c3c9ad15e75742734b1ed22fbde"}
Nov 03 08:14:15.690430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.690416    6430 scope.go:122] "RemoveContainer" containerID="503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a"
Nov 03 08:14:15.690695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.690679    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"
Nov 03 08:14:15.835641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.835109    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"]
Nov 03 08:14:15.850647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.850615    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e255cbee-6778ddc7cf-jhvp7"]
Nov 03 08:14:15.851041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.851024    6430 scope.go:122] "RemoveContainer" containerID="503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a"
Nov 03 08:14:15.851829 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:15.851752    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a\": container with ID starting with 503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a not found: ID does not exist" containerID="503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a"
Nov 03 08:14:15.851829 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:15.851789    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a"} err="failed to get container status \"503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a\": rpc error: code = NotFound desc = could not find container \"503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a\": container with ID starting with 503244b540131f8a9a4162a900c6de74f7306b0e5b83f41589d1f68a7010286a not found: ID does not exist"
Nov 03 08:14:16.174187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:16.174155    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:16.621996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:16.621966    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6490b36f-7cd0-4dfa-8191-5fb41fb632d9" path="/var/lib/kubelet/pods/6490b36f-7cd0-4dfa-8191-5fb41fb632d9/volumes"
Nov 03 08:14:17.061179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:17.061149    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:17.844867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:17.844836    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:18.756321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:18.756282    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:19.377539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:19.377505    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:19.494894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:19.494856    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:19.688146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:19.683161    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:20.750565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:20.750532    6430 generic.go:356] "Generic (PLEG): container finished" podID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerID="ace50e7b444d65ae218b150a8bd7b9dedcaf67307a216efd560baeabbef15f6f" exitCode=0
Nov 03 08:14:20.751041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:20.750762    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" event={"ID":"b7019add-ed7c-4cc8-bcfc-0492666161d3","Type":"ContainerDied","Data":"ace50e7b444d65ae218b150a8bd7b9dedcaf67307a216efd560baeabbef15f6f"}
Nov 03 08:14:20.753335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:20.753299    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:20.753465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:20.753343    6430 generic.go:356] "Generic (PLEG): container finished" podID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerID="acc970c734879c26f77be2f6f2925204f8bef852e5022be204c12519c582de0b" exitCode=0
Nov 03 08:14:20.753515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:20.753457    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" event={"ID":"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137","Type":"ContainerDied","Data":"acc970c734879c26f77be2f6f2925204f8bef852e5022be204c12519c582de0b"}
Nov 03 08:14:20.931650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:20.931228    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:21.063167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.063133    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:21.141187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.130904    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:21.141187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.132692    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istiod-ca-cert\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.141187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.132831    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-certs\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.141187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.132981    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-data\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.141187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.133056    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-credential-socket\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.141187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.133176    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-podinfo\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.154167    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-certs" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.154307    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-token\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-token\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.154478    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-kube-api-access-25kcs\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-kube-api-access-25kcs\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.154538    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-envoy\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.154587    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-socket\") pod \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\" (UID: \"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137\") "
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.155528    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.162446    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-credential-socket" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.174252    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istiod-ca-cert" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:21.190169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.188176    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-data" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.194240 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.194185    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-socket" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.293730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.263832    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.293730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.263870    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.293730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.263888    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.293730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.263902    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.327161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.321373    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:21.347085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.346717    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-podinfo" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:21.366003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.365946    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.383518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.383460    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-kube-api-access-25kcs" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "kube-api-access-25kcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:21.390501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.390441    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-token" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:21.438465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.436730    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-envoy" pod "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" (UID: "dfc3e0dd-c754-4c11-8bb9-a0e66fa97137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.494374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.494282    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-socket\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.494908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.494799    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-socket" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.499406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.499386    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7019add-ed7c-4cc8-bcfc-0492666161d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7019add-ed7c-4cc8-bcfc-0492666161d3-istiod-ca-cert\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.518636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.518609    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-data\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.518497    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7019add-ed7c-4cc8-bcfc-0492666161d3-istiod-ca-cert" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.522033    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-data" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.528544    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-kube-api-access-psf9d\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-kube-api-access-psf9d\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.528607    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-podinfo\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.528722    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-envoy\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.528775    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-token\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-token\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.528829    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-certs\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.529045    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-credential-socket\") pod \"b7019add-ed7c-4cc8-bcfc-0492666161d3\" (UID: \"b7019add-ed7c-4cc8-bcfc-0492666161d3\") "
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.531220    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.531248    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.531265    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7019add-ed7c-4cc8-bcfc-0492666161d3-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.531293    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.531306    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.531320    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kcs\" (UniqueName: \"kubernetes.io/projected/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137-kube-api-access-25kcs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.531891    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-credential-socket" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.541123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.537132    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-certs" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.545970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.545922    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-podinfo" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:21.550010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.549950    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-token" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:21.551924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.551887    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-kube-api-access-psf9d" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "kube-api-access-psf9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:21.553512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.553472    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-envoy" pod "b7019add-ed7c-4cc8-bcfc-0492666161d3" (UID: "b7019add-ed7c-4cc8-bcfc-0492666161d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:21.633771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.633735    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psf9d\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-kube-api-access-psf9d\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.633771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.633773    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.633984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.633788    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.633984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.633803    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7019add-ed7c-4cc8-bcfc-0492666161d3-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.633984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.633817    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.633984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.633832    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7019add-ed7c-4cc8-bcfc-0492666161d3-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:21.867138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.866281    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:21.893088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.875051    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g" event={"ID":"b7019add-ed7c-4cc8-bcfc-0492666161d3","Type":"ContainerDied","Data":"fa3664600adbee1e4ce19296316658ad38b3b65564d5194727c2555c68bb89df"}
Nov 03 08:14:21.893088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.875145    6430 scope.go:122] "RemoveContainer" containerID="ace50e7b444d65ae218b150a8bd7b9dedcaf67307a216efd560baeabbef15f6f"
Nov 03 08:14:21.893088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:21.875290    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"
Nov 03 08:14:22.114916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.114876    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb" event={"ID":"dfc3e0dd-c754-4c11-8bb9-a0e66fa97137","Type":"ContainerDied","Data":"ed69582be904ddd5bba3887d4d311fd11beb10f174aff684154a448495c8cd8e"}
Nov 03 08:14:22.128234 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.117354    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"
Nov 03 08:14:22.229396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.229360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:22.269079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.269003    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"]
Nov 03 08:14:22.338086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.330234    6430 scope.go:122] "RemoveContainer" containerID="acc970c734879c26f77be2f6f2925204f8bef852e5022be204c12519c582de0b"
Nov 03 08:14:22.405968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.405851    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:22.592333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.592299    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-qd67g"]
Nov 03 08:14:22.813587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.813539    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" path="/var/lib/kubelet/pods/b7019add-ed7c-4cc8-bcfc-0492666161d3/volumes"
Nov 03 08:14:22.814309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.814284    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"]
Nov 03 08:14:22.814498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.814486    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-zjzpb"]
Nov 03 08:14:22.818305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:22.818006    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:23.956090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:23.951966    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:24.644592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:24.644559    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" path="/var/lib/kubelet/pods/dfc3e0dd-c754-4c11-8bb9-a0e66fa97137/volumes"
Nov 03 08:14:24.937034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:24.936994    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:25.748187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:25.748149    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:25.858897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:25.858859    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:25.971563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:25.971516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:26.604716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:26.604678    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:26.971426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:26.971379    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:27.089575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:27.089522    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:27.225804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:27.225697    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:27.377088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:27.376254    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:14:27.512934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:27.512842    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:14:27.553463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:27.553422    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:14:27.623604 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:27.623564    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:28.283251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:28.278216    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:28.443546 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:28.443503    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:14:29.236290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:29.236255    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:30.000438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:30.000366    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:30.194968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:30.194933    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:30.362366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:30.362282    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:30.902055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:30.902026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:31.050830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.050789    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:31.205120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.205082    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:31.327950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.327921    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:31.713735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.713698    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:31.723885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.723844    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"]
Nov 03 08:14:31.724113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.724085    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy" containerID="cri-o://041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3" gracePeriod=30
Nov 03 08:14:31.830396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.830360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:31.890123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.887970    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"]
Nov 03 08:14:31.890123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:31.888320    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy" containerID="cri-o://38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd" gracePeriod=30
Nov 03 08:14:32.202393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:32.157081    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:32.577131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:32.576603    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"]
Nov 03 08:14:32.638156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:32.638092    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:33.030458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:33.030421    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:33.961236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:33.961198    6430 patch_prober.go:29] interesting pod/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:14:33.961827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:33.961790    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:14:33.973300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:33.973274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:34.655984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:34.655946    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:35.518362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.518274    6430 generic.go:356] "Generic (PLEG): container finished" podID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerID="d29585134e5b7b1446dfffd90579c9a48444dad017e89d6f44506c503a69db70" exitCode=0
Nov 03 08:14:35.518362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.518349    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" event={"ID":"d62d7575-8abc-452e-ba8d-b0965e6f683e","Type":"ContainerDied","Data":"d29585134e5b7b1446dfffd90579c9a48444dad017e89d6f44506c503a69db70"}
Nov 03 08:14:35.518858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.518843    6430 scope.go:122] "RemoveContainer" containerID="d29585134e5b7b1446dfffd90579c9a48444dad017e89d6f44506c503a69db70"
Nov 03 08:14:35.520757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.520725    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:35.520857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.520811    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:35.520896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.520853    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:35.598824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.598798    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:35.895235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:35.895201    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:36.027194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:36.027148    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:36.536787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:36.536755    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" event={"ID":"d62d7575-8abc-452e-ba8d-b0965e6f683e","Type":"ContainerStarted","Data":"8e78bc2774c8c1e712558d59b2447306638ca7bef149f7a6439da15c5c27d99e"}
Nov 03 08:14:36.649564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:36.649533    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:36.974214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:36.974093    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:14:36.974772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:36.974436    6430 patch_prober.go:29] interesting pod/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 08:14:36.974772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:36.974492    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 08:14:37.080470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.080420    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:14:37.158019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.157989    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:37.256726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.256696    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-socket\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.256861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.256796    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c4fcc3-a069-4c7c-9d32-bf475a418747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c4fcc3-a069-4c7c-9d32-bf475a418747-istiod-ca-cert\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.256861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.256829    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-envoy\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.256971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.256956    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-credential-socket\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.257010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.256988    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-kube-api-access-lbnqw\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-kube-api-access-lbnqw\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.257053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.257027    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-data\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.257157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.257093    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-token\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-token\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.257347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.257328    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-podinfo\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.257429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.257407    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-certs\") pod \"34c4fcc3-a069-4c7c-9d32-bf475a418747\" (UID: \"34c4fcc3-a069-4c7c-9d32-bf475a418747\") "
Nov 03 08:14:37.257692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.257552    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-credential-socket" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.258102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.257950    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-data" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.258405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.258179    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c4fcc3-a069-4c7c-9d32-bf475a418747-istiod-ca-cert" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:37.258764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.258743    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-certs" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.259632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.259330    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c4fcc3-a069-4c7c-9d32-bf475a418747-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.259632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.259358    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.259632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.259375    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.259632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.259571    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-socket" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.261829 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.261791    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-token" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:37.262574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.262549    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-envoy" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.263133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.262909    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-podinfo" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:37.263211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.263186    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-kube-api-access-lbnqw" pod "34c4fcc3-a069-4c7c-9d32-bf475a418747" (UID: "34c4fcc3-a069-4c7c-9d32-bf475a418747"). InnerVolumeSpecName "kube-api-access-lbnqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:37.265824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.265804    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:14:37.348139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.344560    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:37.374088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.367461    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.374088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.367501    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.374088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.367519    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.374088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.367532    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.374088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.367547    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnqw\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-kube-api-access-lbnqw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.374088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.367569    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c4fcc3-a069-4c7c-9d32-bf475a418747-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.495598    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-certs\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.495692    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-podinfo\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.495734    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-envoy\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.495833    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istiod-ca-cert\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.495874    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-socket\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.496009    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.496040    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-data\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.496182    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-credential-socket\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.503232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.496257    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-kube-api-access-62kjb\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-kube-api-access-62kjb\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.539142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.535154    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-certs" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.539142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.535633    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istiod-ca-cert" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:37.539142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.535921    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-data" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.555111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.553202    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-credential-socket" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.555111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.553458    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-socket" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.606803    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.607372    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\") pod \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\" (UID: \"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48\") "
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:14:37.607485    6430 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48/volumes/kubernetes.io~projected/istio-token
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.607525    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.608320    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.608341    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.608356    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.608367    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.608379    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.618151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.608390    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.658104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.634298    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-podinfo" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:37.692100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.684704    6430 generic.go:356] "Generic (PLEG): container finished" podID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerID="38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd" exitCode=0
Nov 03 08:14:37.692100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.684922    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" event={"ID":"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48","Type":"ContainerDied","Data":"38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd"}
Nov 03 08:14:37.692100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.685222    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2" event={"ID":"2366709e-fe4d-4cbd-8aa7-ec0f618b4a48","Type":"ContainerDied","Data":"7f36810fbf30812dd5d14b3afa64523571b9ead62b07b23f420d6519da3ffb22"}
Nov 03 08:14:37.692100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.685280    6430 scope.go:122] "RemoveContainer" containerID="38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd"
Nov 03 08:14:37.692100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.685466    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"
Nov 03 08:14:37.733103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.717265    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.743804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.741042    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-kube-api-access-62kjb" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "kube-api-access-62kjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:37.743804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.742523    6430 generic.go:356] "Generic (PLEG): container finished" podID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerID="041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3" exitCode=0
Nov 03 08:14:37.743804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.743665    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"
Nov 03 08:14:37.759097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.752847    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" event={"ID":"34c4fcc3-a069-4c7c-9d32-bf475a418747","Type":"ContainerDied","Data":"041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3"}
Nov 03 08:14:37.759097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.752917    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl" event={"ID":"34c4fcc3-a069-4c7c-9d32-bf475a418747","Type":"ContainerDied","Data":"45825be3f66b8f70ed577c5654466e76008720b5bcc5e908369c664d31f92191"}
Nov 03 08:14:37.774098 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:37.764120    6430 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c4fcc3_a069_4c7c_9d32_bf475a418747.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:37.774261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.774237    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-envoy" pod "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" (UID: "2366709e-fe4d-4cbd-8aa7-ec0f618b4a48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:37.789629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.789596    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"]
Nov 03 08:14:37.792712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.792690    6430 scope.go:122] "RemoveContainer" containerID="38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd"
Nov 03 08:14:37.793150 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:37.793121    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd\": container with ID starting with 38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd not found: ID does not exist" containerID="38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd"
Nov 03 08:14:37.793220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.793188    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd"} err="failed to get container status \"38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd\": rpc error: code = NotFound desc = could not find container \"38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd\": container with ID starting with 38de9590e7b4f07b3f9caef7d62579dd38b8dc40b897017e1160c6ade7a328fd not found: ID does not exist"
Nov 03 08:14:37.793220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.793214    6430 scope.go:122] "RemoveContainer" containerID="041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3"
Nov 03 08:14:37.830535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.820545    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62kjb\" (UniqueName: \"kubernetes.io/projected/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-kube-api-access-62kjb\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.830535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.820574    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:37.830535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.825785    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-j4zrl"]
Nov 03 08:14:37.830535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.827205    6430 scope.go:122] "RemoveContainer" containerID="041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3"
Nov 03 08:14:37.830535 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:37.827604    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3\": container with ID starting with 041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3 not found: ID does not exist" containerID="041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3"
Nov 03 08:14:37.830535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.827629    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3"} err="failed to get container status \"041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3\": rpc error: code = NotFound desc = could not find container \"041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3\": container with ID starting with 041c952c27fe8ef4278d788c9fa154c729f3072d19b14b55a0243b051f8c09b3 not found: ID does not exist"
Nov 03 08:14:37.966602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.966573    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:37.975171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.975143    6430 patch_prober.go:29] interesting pod/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 08:14:37.975300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:37.975191    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 08:14:38.012997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.012962    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"]
Nov 03 08:14:38.036800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.036767    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-fx7t2"]
Nov 03 08:14:38.089785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.089757    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:38.243142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.242604    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:38.693296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.693260    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" path="/var/lib/kubelet/pods/2366709e-fe4d-4cbd-8aa7-ec0f618b4a48/volumes"
Nov 03 08:14:38.694989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.694954    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" path="/var/lib/kubelet/pods/34c4fcc3-a069-4c7c-9d32-bf475a418747/volumes"
Nov 03 08:14:38.998139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.997119    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:39.007095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.999438    6430 patch_prober.go:29] interesting pod/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 08:14:39.007095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:38.999503    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 08:14:39.731534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:39.731474    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:39.975804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:39.975768    6430 patch_prober.go:29] interesting pod/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 08:14:39.976082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:39.976037    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 08:14:40.455273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:40.455238    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:40.974238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:40.974206    6430 patch_prober.go:29] interesting pod/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 08:14:40.974650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:40.974265    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 08:14:41.163931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:41.163894    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:41.736094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:41.736033    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:41.979128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:41.978619    6430 patch_prober.go:29] interesting pod/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 08:14:41.979128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:41.978702    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 08:14:41.986087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:41.984523    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:42.123166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:42.116711    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:42.665567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:42.665531    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:42.979529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:42.979501    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:14:42.979919 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:42.979855    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:14:42.980532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:42.980518    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:14:43.464504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.464472    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:43.770719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.770630    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:43.895911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.895869    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"]
Nov 03 08:14:43.897360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897335    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy"
Nov 03 08:14:43.897360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897360    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897377    6430 cpu_manager.go:414] "containerMap: removing container" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897385    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897402    6430 cpu_manager.go:414] "containerMap: removing container" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897411    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897461    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897470    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897483    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6490b36f-7cd0-4dfa-8191-5fb41fb632d9" containerName="payload-processing"
Nov 03 08:14:43.897545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897491    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6490b36f-7cd0-4dfa-8191-5fb41fb632d9" containerName="payload-processing"
Nov 03 08:14:43.897823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897681    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6490b36f-7cd0-4dfa-8191-5fb41fb632d9" containerName="payload-processing"
Nov 03 08:14:43.897823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897702    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2366709e-fe4d-4cbd-8aa7-ec0f618b4a48" containerName="istio-proxy"
Nov 03 08:14:43.897823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897724    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc3e0dd-c754-4c11-8bb9-a0e66fa97137" containerName="istio-proxy"
Nov 03 08:14:43.897823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897736    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c4fcc3-a069-4c7c-9d32-bf475a418747" containerName="istio-proxy"
Nov 03 08:14:43.897823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.897752    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7019add-ed7c-4cc8-bcfc-0492666161d3" containerName="istio-proxy"
Nov 03 08:14:43.903733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.903706    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.911966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.911933    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb887217-openshift-default-dockercfg-6fvp6\""
Nov 03 08:14:43.914368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.914337    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"]
Nov 03 08:14:43.981483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.981450    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-data\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.981884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.981526    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5231c00e-b16d-40a8-9dbe-76f1509d758b-istiod-ca-cert\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.981884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.981611    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-envoy\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.981884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.981681    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-token\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.981884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.981798    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-podinfo\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.981884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.981869    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-certs\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.982178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.981905    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2cq\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-kube-api-access-fl2cq\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.982178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.982128    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-credential-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:43.982249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:43.982178    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.083990    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-certs\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084094    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2cq\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-kube-api-access-fl2cq\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084155    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-credential-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084206    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084315    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-data\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084413    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5231c00e-b16d-40a8-9dbe-76f1509d758b-istiod-ca-cert\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084479    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-envoy\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084525    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-token\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084538    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-certs\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084562    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-credential-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084628    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.084782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.084645    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-podinfo\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.085030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.085006    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-data\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.085476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.085453    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5231c00e-b16d-40a8-9dbe-76f1509d758b-istiod-ca-cert\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.087004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.086975    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-envoy\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.087457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.087433    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-podinfo\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.093245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.093222    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-token\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.093440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.093416    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2cq\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-kube-api-access-fl2cq\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.208885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.208842    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:44.221920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.221884    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:44.453099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.451956    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:44.605112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:44.600049    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:45.118119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:45.118072    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"]
Nov 03 08:14:45.119839 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:14:45.119818    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5231c00e_b16d_40a8_9dbe_76f1509d758b.slice/crio-fecaa1f326f6a87b8f1b7d90987a59bccb6f75d65297e4d08e2a577557b76e13 WatchSource:0}: Error finding container fecaa1f326f6a87b8f1b7d90987a59bccb6f75d65297e4d08e2a577557b76e13: Status 404 returned error can't find the container with id fecaa1f326f6a87b8f1b7d90987a59bccb6f75d65297e4d08e2a577557b76e13
Nov 03 08:14:45.122651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:45.122621    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:45.122741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:45.122684    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:45.122741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:45.122712    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:14:45.520451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:45.520415    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:46.040090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:46.039850    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" event={"ID":"5231c00e-b16d-40a8-9dbe-76f1509d758b","Type":"ContainerStarted","Data":"fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420"}
Nov 03 08:14:46.040090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:46.039920    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" event={"ID":"5231c00e-b16d-40a8-9dbe-76f1509d758b","Type":"ContainerStarted","Data":"fecaa1f326f6a87b8f1b7d90987a59bccb6f75d65297e4d08e2a577557b76e13"}
Nov 03 08:14:46.073090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:46.068707    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podStartSLOduration=3.068688865 podStartE2EDuration="3.068688865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:46.063449279 +0000 UTC m=+1235.566284804" watchObservedRunningTime="2026-11-03 08:14:46.068688865 +0000 UTC m=+1235.571524383"
Nov 03 08:14:46.228092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:46.226779    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:46.229611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:46.229578    6430 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:14:46.229837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:46.229802    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 08:14:46.339972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:46.339877    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:47.222478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:47.222442    6430 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:14:47.222661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:47.222511    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 08:14:47.336908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:47.336873    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:47.623742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:47.623644    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main" containerID="cri-o://03bafd0fe4b55e56976f26ea8e2097648596b3fdaee1e84a9d4540902cccfd23" gracePeriod=45
Nov 03 08:14:47.879646 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:47.879561    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:48.139672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.139589    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:48.145927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.145901    6430 generic.go:356] "Generic (PLEG): container finished" podID="13a29667-3653-485c-8cac-3471c5a15ba9" containerID="03bafd0fe4b55e56976f26ea8e2097648596b3fdaee1e84a9d4540902cccfd23" exitCode=0
Nov 03 08:14:48.146846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.146821    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" event={"ID":"13a29667-3653-485c-8cac-3471c5a15ba9","Type":"ContainerDied","Data":"03bafd0fe4b55e56976f26ea8e2097648596b3fdaee1e84a9d4540902cccfd23"}
Nov 03 08:14:48.173995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.173971    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:14:48.230917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.230854    6430 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:14:48.231146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.230974    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 08:14:48.386120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.376086    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:48.464087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.456330    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-dshm\") pod \"13a29667-3653-485c-8cac-3471c5a15ba9\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") "
Nov 03 08:14:48.464087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.456687    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-home\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-home\") pod \"13a29667-3653-485c-8cac-3471c5a15ba9\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") "
Nov 03 08:14:48.571094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.563494    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-tmp-dir\") pod \"13a29667-3653-485c-8cac-3471c5a15ba9\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") "
Nov 03 08:14:48.571818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.571792    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-model-cache\") pod \"13a29667-3653-485c-8cac-3471c5a15ba9\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") "
Nov 03 08:14:48.582706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.582680    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a29667-3653-485c-8cac-3471c5a15ba9-kube-api-access-c2sxs\" (UniqueName: \"kubernetes.io/projected/13a29667-3653-485c-8cac-3471c5a15ba9-kube-api-access-c2sxs\") pod \"13a29667-3653-485c-8cac-3471c5a15ba9\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") "
Nov 03 08:14:48.605034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.605002    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13a29667-3653-485c-8cac-3471c5a15ba9-tls-certs\" (UniqueName: \"kubernetes.io/secret/13a29667-3653-485c-8cac-3471c5a15ba9-tls-certs\") pod \"13a29667-3653-485c-8cac-3471c5a15ba9\" (UID: \"13a29667-3653-485c-8cac-3471c5a15ba9\") "
Nov 03 08:14:48.615083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.495279    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-home" pod "13a29667-3653-485c-8cac-3471c5a15ba9" (UID: "13a29667-3653-485c-8cac-3471c5a15ba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:48.676130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.577703    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-tmp-dir" pod "13a29667-3653-485c-8cac-3471c5a15ba9" (UID: "13a29667-3653-485c-8cac-3471c5a15ba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:48.676356 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.582217    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-model-cache" pod "13a29667-3653-485c-8cac-3471c5a15ba9" (UID: "13a29667-3653-485c-8cac-3471c5a15ba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:48.676449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.604211    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-dshm" pod "13a29667-3653-485c-8cac-3471c5a15ba9" (UID: "13a29667-3653-485c-8cac-3471c5a15ba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:48.676529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.626489    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a29667-3653-485c-8cac-3471c5a15ba9-kube-api-access-c2sxs" pod "13a29667-3653-485c-8cac-3471c5a15ba9" (UID: "13a29667-3653-485c-8cac-3471c5a15ba9"). InnerVolumeSpecName "kube-api-access-c2sxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:48.676605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.673677    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a29667-3653-485c-8cac-3471c5a15ba9-tls-certs" pod "13a29667-3653-485c-8cac-3471c5a15ba9" (UID: "13a29667-3653-485c-8cac-3471c5a15ba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:14:48.708496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.708462    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:48.708496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.708497    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2sxs\" (UniqueName: \"kubernetes.io/projected/13a29667-3653-485c-8cac-3471c5a15ba9-kube-api-access-c2sxs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:48.708730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.708511    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a29667-3653-485c-8cac-3471c5a15ba9-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:48.708730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.708523    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:48.708730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.708534    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:48.708730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.708546    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a29667-3653-485c-8cac-3471c5a15ba9-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:48.971747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:48.971660    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:14:49.162605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.162574    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"
Nov 03 08:14:49.162774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.162610    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2" event={"ID":"13a29667-3653-485c-8cac-3471c5a15ba9","Type":"ContainerDied","Data":"a063d7b79ee9a4a5e521ef53aaf8d1f9b3ec95ce5b8ba2c5f43060213cce534a"}
Nov 03 08:14:49.162774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.162674    6430 scope.go:122] "RemoveContainer" containerID="03bafd0fe4b55e56976f26ea8e2097648596b3fdaee1e84a9d4540902cccfd23"
Nov 03 08:14:49.179637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.179612    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"]
Nov 03 08:14:49.183502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.183480    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cb887217/e2e-lifecycle-model-cb887217-kserve-6dd4c5777c-bj2k2"]
Nov 03 08:14:49.223014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.222941    6430 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:14:49.223014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.222999    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 08:14:49.356235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.356204    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:49.549650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.540767    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:49.698085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:49.688146    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:50.223505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:50.223467    6430 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:14:50.223688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:50.223549    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 08:14:50.634559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:50.634467    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" path="/var/lib/kubelet/pods/13a29667-3653-485c-8cac-3471c5a15ba9/volumes"
Nov 03 08:14:50.735712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:50.735685    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:51.229500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:51.229467    6430 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:14:51.229896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:51.229867    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 08:14:51.521157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:51.513632    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:52.281381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:52.281355    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:52.282191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:52.282164    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:52.283137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:52.283119    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:14:52.501137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:52.480555    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:53.071200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.071168    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"]
Nov 03 08:14:53.071944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.071929    6430 cpu_manager.go:414] "containerMap: removing container" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main"
Nov 03 08:14:53.072001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.071946    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main"
Nov 03 08:14:53.072052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.072043    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a29667-3653-485c-8cac-3471c5a15ba9" containerName="main"
Nov 03 08:14:53.077990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.077967    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.085913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.085884    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"]
Nov 03 08:14:53.093386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.093357    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-950e06-serving-cert\""
Nov 03 08:14:53.123484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.123448    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-openshift-service-ca\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.123692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.123613    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-maas-api-tls\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.123692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.123668    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-metrics-certs\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.123819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.123792    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqcm2\" (UniqueName: \"kubernetes.io/projected/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-kube-api-access-jqcm2\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.240850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.235942    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-maas-api-tls\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.240850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.236034    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-metrics-certs\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.240850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.236118    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqcm2\" (UniqueName: \"kubernetes.io/projected/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-kube-api-access-jqcm2\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.240850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.236277    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-openshift-service-ca\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.240850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.237858    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-openshift-service-ca\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.266746 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.253643    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:53.279226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.267219    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-metrics-certs\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.279226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.270330    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-maas-api-tls\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.325152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.323021    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqcm2\" (UniqueName: \"kubernetes.io/projected/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-kube-api-access-jqcm2\") pod \"maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.374921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.372757    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:53.403977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.403949    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:53.867845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.861496    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"]
Nov 03 08:14:53.887569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:53.884578    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:14:54.029220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:54.029191    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:54.269742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:54.269708    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp" event={"ID":"8461f6e4-df91-4a0b-9cc4-0bda70a26a76","Type":"ContainerStarted","Data":"bad353ba305b1e263635b501a12fb3e8daf69e5fddf823ca0135f1dc2cb22d3d"}
Nov 03 08:14:54.746828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:54.746794    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:55.253411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:55.253379    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:55.278045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:55.278013    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp" event={"ID":"8461f6e4-df91-4a0b-9cc4-0bda70a26a76","Type":"ContainerStarted","Data":"b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2"}
Nov 03 08:14:55.278260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:55.278089    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:14:55.296769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:55.296715    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp" podStartSLOduration=1.8037020240000001 podStartE2EDuration="2.296701394s" totalImagesPullingTime="492.99937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:55.293400252 +0000 UTC m=+1244.796235767" watchObservedRunningTime="2026-11-03 08:14:55.296701394 +0000 UTC m=+1244.799536907"
Nov 03 08:14:55.465920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:55.465881    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:55.734733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:55.734659    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:56.131891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:56.127922    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:56.605581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:56.605549    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:57.742856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:57.742818    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:58.390528 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:14:58.390485    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:14:58.530352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:58.530314    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:59.187360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:59.187325    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"]
Nov 03 08:14:59.307491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:59.307459    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:59.530858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:59.527537    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:59.664630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:59.664579    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:14:59.803666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:14:59.803576    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:00.031570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.031522    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:00.197729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.197693    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"]
Nov 03 08:15:00.397456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.395743    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"]
Nov 03 08:15:00.397939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.397914    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.464126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.463867    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.464126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.463939    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbz87\" (UniqueName: \"kubernetes.io/projected/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-kube-api-access-sbz87\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.478228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.478161    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:15:00.482217 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.482194    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"]
Nov 03 08:15:00.482403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.482393    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"]
Nov 03 08:15:00.483588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.483568    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:00.489190 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.489151    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"]
Nov 03 08:15:00.489301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.489268    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.495218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.495199    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"]
Nov 03 08:15:00.499019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.498998    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"]
Nov 03 08:15:00.504283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.504261    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.507477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.507453    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"]
Nov 03 08:15:00.511854 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.511826    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:00.519144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.518909    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"]
Nov 03 08:15:00.520963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.519298    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:00.524425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.524401    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"]
Nov 03 08:15:00.565779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.565743    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.565955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.565806    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbz87\" (UniqueName: \"kubernetes.io/projected/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-kube-api-access-sbz87\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.566446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.566422    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.573972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.573940    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbz87\" (UniqueName: \"kubernetes.io/projected/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-kube-api-access-sbz87\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.667585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.667542    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714d107-1408-451e-be96-16017082f797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g\" (UID: \"5714d107-1408-451e-be96-16017082f797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.667803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.667610    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvngh\" (UniqueName: \"kubernetes.io/projected/3d7fad21-fe12-4d47-911d-c036b42ea3ad-kube-api-access-pvngh\") pod \"maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.667803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.667758    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7fad21-fe12-4d47-911d-c036b42ea3ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.667921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.667834    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f06d813-d72f-4516-90b1-55419f11af20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:00.667961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.667931    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbsr\" (UniqueName: \"kubernetes.io/projected/7f06d813-d72f-4516-90b1-55419f11af20-kube-api-access-qzbsr\") pod \"maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:00.668012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.667999    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wsz\" (UniqueName: \"kubernetes.io/projected/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-kube-api-access-n4wsz\") pod \"maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:00.668206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.668181    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:00.668284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.668219    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfxs\" (UniqueName: \"kubernetes.io/projected/5714d107-1408-451e-be96-16017082f797-kube-api-access-mmfxs\") pod \"maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g\" (UID: \"5714d107-1408-451e-be96-16017082f797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.764493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.764406    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.770153    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.770213    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfxs\" (UniqueName: \"kubernetes.io/projected/5714d107-1408-451e-be96-16017082f797-kube-api-access-mmfxs\") pod \"maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g\" (UID: \"5714d107-1408-451e-be96-16017082f797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.770306    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714d107-1408-451e-be96-16017082f797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g\" (UID: \"5714d107-1408-451e-be96-16017082f797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.770361    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvngh\" (UniqueName: \"kubernetes.io/projected/3d7fad21-fe12-4d47-911d-c036b42ea3ad-kube-api-access-pvngh\") pod \"maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.772916    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.774357    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714d107-1408-451e-be96-16017082f797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g\" (UID: \"5714d107-1408-451e-be96-16017082f797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.774452    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7fad21-fe12-4d47-911d-c036b42ea3ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.770200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.774624    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f06d813-d72f-4516-90b1-55419f11af20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:00.809096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.804042    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7fad21-fe12-4d47-911d-c036b42ea3ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.809096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.805962    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvngh\" (UniqueName: \"kubernetes.io/projected/3d7fad21-fe12-4d47-911d-c036b42ea3ad-kube-api-access-pvngh\") pod \"maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.848002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.847959    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:00.854539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.854485    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbsr\" (UniqueName: \"kubernetes.io/projected/7f06d813-d72f-4516-90b1-55419f11af20-kube-api-access-qzbsr\") pod \"maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:00.861126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.859905    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f06d813-d72f-4516-90b1-55419f11af20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:00.869117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.867715    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wsz\" (UniqueName: \"kubernetes.io/projected/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-kube-api-access-n4wsz\") pod \"maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:00.869117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.868577    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:00.885126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.884433    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfxs\" (UniqueName: \"kubernetes.io/projected/5714d107-1408-451e-be96-16017082f797-kube-api-access-mmfxs\") pod \"maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g\" (UID: \"5714d107-1408-451e-be96-16017082f797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.893098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.889990    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:00.919967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.919930    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbsr\" (UniqueName: \"kubernetes.io/projected/7f06d813-d72f-4516-90b1-55419f11af20-kube-api-access-qzbsr\") pod \"maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:00.959099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.957422    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wsz\" (UniqueName: \"kubernetes.io/projected/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-kube-api-access-n4wsz\") pod \"maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:00.984420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:00.983460    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:01.102318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.102276    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"]
Nov 03 08:15:01.119303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.117463    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:01.123834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.122773    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"]
Nov 03 08:15:01.123834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.122871    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.144107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.132287    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:01.145797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.145506    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:01.231377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.214181    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e0f576\""
Nov 03 08:15:01.231377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.214263    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e0f576-dockercfg-8tn7p\""
Nov 03 08:15:01.311327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.294193    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:01.311327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.295381    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fd0430-b072-458b-91fa-8a1459789c95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.311327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.295626    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgcq6\" (UniqueName: \"kubernetes.io/projected/89fd0430-b072-458b-91fa-8a1459789c95-kube-api-access-fgcq6\") pod \"payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.600315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.599228    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:15:01.673418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.669742    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgcq6\" (UniqueName: \"kubernetes.io/projected/89fd0430-b072-458b-91fa-8a1459789c95-kube-api-access-fgcq6\") pod \"payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.673418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.669940    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fd0430-b072-458b-91fa-8a1459789c95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.685011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.677117    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"]
Nov 03 08:15:01.723840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.723619    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgcq6\" (UniqueName: \"kubernetes.io/projected/89fd0430-b072-458b-91fa-8a1459789c95-kube-api-access-fgcq6\") pod \"payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.736292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.736220    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fd0430-b072-458b-91fa-8a1459789c95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.769463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.769438    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:01.977296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.977256    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"]
Nov 03 08:15:01.983013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.982978    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:01.987236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:01.985631    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"]
Nov 03 08:15:01.991714 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:01.991675    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9e252c_56a1_483a_af31_b5a7eb2d1ff5.slice/crio-148876e80b0d4096f11b2f9c3fda91b60a39e680cde70cb8a2e56eada5a10e2d WatchSource:0}: Error finding container 148876e80b0d4096f11b2f9c3fda91b60a39e680cde70cb8a2e56eada5a10e2d: Status 404 returned error can't find the container with id 148876e80b0d4096f11b2f9c3fda91b60a39e680cde70cb8a2e56eada5a10e2d
Nov 03 08:15:02.209454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.209422    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"]
Nov 03 08:15:02.218818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.218786    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"]
Nov 03 08:15:02.333587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.333564    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"]
Nov 03 08:15:02.402888 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:02.402845    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fd0430_b072_458b_91fa_8a1459789c95.slice/crio-db528f663d22bf9f97a943121a23d4994b70aa4d12979727b4d79c6561885e6c WatchSource:0}: Error finding container db528f663d22bf9f97a943121a23d4994b70aa4d12979727b4d79c6561885e6c: Status 404 returned error can't find the container with id db528f663d22bf9f97a943121a23d4994b70aa4d12979727b4d79c6561885e6c
Nov 03 08:15:02.740769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.740733    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" event={"ID":"3d7fad21-fe12-4d47-911d-c036b42ea3ad","Type":"ContainerStarted","Data":"0a1a4fef2f84b57ed7de0c0f719fa4575a6ff7ca94b2a09c613799dcf12d4538"}
Nov 03 08:15:02.740895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.740794    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" event={"ID":"3d7fad21-fe12-4d47-911d-c036b42ea3ad","Type":"ContainerStarted","Data":"08513d7b8645af5a014f6fcba14c39ff1c8b349cafe5d61a0601fe43dd0e0e55"}
Nov 03 08:15:02.742303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.742276    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" event={"ID":"7f06d813-d72f-4516-90b1-55419f11af20","Type":"ContainerStarted","Data":"59425eba77a26af4a94a9e9d3f222fba9d3703ec51c29468dca8341418ccc260"}
Nov 03 08:15:02.744304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.744275    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" event={"ID":"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5","Type":"ContainerStarted","Data":"fd6fc3f9026e8faa66a3b56d579e8ef847b44e3fc32b061b61435ba7d1a27390"}
Nov 03 08:15:02.744418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.744332    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" event={"ID":"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5","Type":"ContainerStarted","Data":"148876e80b0d4096f11b2f9c3fda91b60a39e680cde70cb8a2e56eada5a10e2d"}
Nov 03 08:15:02.745689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.745663    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" event={"ID":"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e","Type":"ContainerStarted","Data":"9f9fc1d52b77c3c4cc6ed37a995b0858396fcb3d78126b1918de4c5451b5dacc"}
Nov 03 08:15:02.747710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.747629    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerStarted","Data":"a0c9177eb35ac64881a9dc9a6813ddb02d9ae154f3e8fd722eb06e91a0f398ed"}
Nov 03 08:15:02.747710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.747681    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerStarted","Data":"3b66f91cf817fe303cd38a66d5a12cceb87782aa4738d29d37b963342f94f24d"}
Nov 03 08:15:02.750170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.750145    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9" event={"ID":"89fd0430-b072-458b-91fa-8a1459789c95","Type":"ContainerStarted","Data":"db528f663d22bf9f97a943121a23d4994b70aa4d12979727b4d79c6561885e6c"}
Nov 03 08:15:02.758881 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.758837    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" podStartSLOduration=2.75880168 podStartE2EDuration="2.75880168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.755304833 +0000 UTC m=+1252.258140348" watchObservedRunningTime="2026-11-03 08:15:02.75880168 +0000 UTC m=+1252.261637193"
Nov 03 08:15:02.771109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.770989    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" podStartSLOduration=2.770974507 podStartE2EDuration="2.770974507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.767817253 +0000 UTC m=+1252.270652770" watchObservedRunningTime="2026-11-03 08:15:02.770974507 +0000 UTC m=+1252.273810022"
Nov 03 08:15:02.782839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.782798    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" podStartSLOduration=2.782784821 podStartE2EDuration="2.782784821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.781649626 +0000 UTC m=+1252.284485142" watchObservedRunningTime="2026-11-03 08:15:02.782784821 +0000 UTC m=+1252.285620380"
Nov 03 08:15:02.871431 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:02.871401    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:03.268581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.268542    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:03.678348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.678307    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:03.820887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.820854    6430 generic.go:356] "Generic (PLEG): container finished" podID="7f06d813-d72f-4516-90b1-55419f11af20" containerID="7a4739992498f95d13b32ece20b498cd5b81287cea294dc9a1458fb92480a46a" exitCode=60
Nov 03 08:15:03.821086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.820993    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" event={"ID":"7f06d813-d72f-4516-90b1-55419f11af20","Type":"ContainerDied","Data":"7a4739992498f95d13b32ece20b498cd5b81287cea294dc9a1458fb92480a46a"}
Nov 03 08:15:03.821866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.821329    6430 scope.go:122] "RemoveContainer" containerID="7a4739992498f95d13b32ece20b498cd5b81287cea294dc9a1458fb92480a46a"
Nov 03 08:15:03.823200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.823174    6430 generic.go:356] "Generic (PLEG): container finished" podID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerID="fd6fc3f9026e8faa66a3b56d579e8ef847b44e3fc32b061b61435ba7d1a27390" exitCode=60
Nov 03 08:15:03.823384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.823260    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" event={"ID":"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5","Type":"ContainerDied","Data":"fd6fc3f9026e8faa66a3b56d579e8ef847b44e3fc32b061b61435ba7d1a27390"}
Nov 03 08:15:03.823608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.823561    6430 scope.go:122] "RemoveContainer" containerID="fd6fc3f9026e8faa66a3b56d579e8ef847b44e3fc32b061b61435ba7d1a27390"
Nov 03 08:15:03.825349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.825293    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerID="5ae1fbf650fd55f5d433f6c316194ab27ac1ccb1991a0d10d80a045fed87cb20" exitCode=60
Nov 03 08:15:03.825349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.825330    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" event={"ID":"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e","Type":"ContainerDied","Data":"5ae1fbf650fd55f5d433f6c316194ab27ac1ccb1991a0d10d80a045fed87cb20"}
Nov 03 08:15:03.825638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.825600    6430 scope.go:122] "RemoveContainer" containerID="5ae1fbf650fd55f5d433f6c316194ab27ac1ccb1991a0d10d80a045fed87cb20"
Nov 03 08:15:03.827748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.827724    6430 generic.go:356] "Generic (PLEG): container finished" podID="5714d107-1408-451e-be96-16017082f797" containerID="a0c9177eb35ac64881a9dc9a6813ddb02d9ae154f3e8fd722eb06e91a0f398ed" exitCode=60
Nov 03 08:15:03.827857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.827762    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerDied","Data":"a0c9177eb35ac64881a9dc9a6813ddb02d9ae154f3e8fd722eb06e91a0f398ed"}
Nov 03 08:15:03.827985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.827969    6430 scope.go:122] "RemoveContainer" containerID="a0c9177eb35ac64881a9dc9a6813ddb02d9ae154f3e8fd722eb06e91a0f398ed"
Nov 03 08:15:03.829375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.829352    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9" event={"ID":"89fd0430-b072-458b-91fa-8a1459789c95","Type":"ContainerStarted","Data":"c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63"}
Nov 03 08:15:03.829477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.829421    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:03.830984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.830962    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerID="0a1a4fef2f84b57ed7de0c0f719fa4575a6ff7ca94b2a09c613799dcf12d4538" exitCode=60
Nov 03 08:15:03.831093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.831005    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" event={"ID":"3d7fad21-fe12-4d47-911d-c036b42ea3ad","Type":"ContainerDied","Data":"0a1a4fef2f84b57ed7de0c0f719fa4575a6ff7ca94b2a09c613799dcf12d4538"}
Nov 03 08:15:03.831359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.831339    6430 scope.go:122] "RemoveContainer" containerID="0a1a4fef2f84b57ed7de0c0f719fa4575a6ff7ca94b2a09c613799dcf12d4538"
Nov 03 08:15:03.901952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:03.901893    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9" podStartSLOduration=2.901874617 podStartE2EDuration="2.901874617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.89940562 +0000 UTC m=+1253.402241137" watchObservedRunningTime="2026-11-03 08:15:03.901874617 +0000 UTC m=+1253.404710133"
Nov 03 08:15:04.188325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.188283    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz" podUID="e24415d0-ef79-46c2-bb7f-4f44d44fea97" containerName="payload-pre-processing" containerID="cri-o://fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a" gracePeriod=25
Nov 03 08:15:04.503974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.503944    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:15:04.613984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.613950    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24415d0-ef79-46c2-bb7f-4f44d44fea97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24415d0-ef79-46c2-bb7f-4f44d44fea97-plugins-config-volume\") pod \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") "
Nov 03 08:15:04.614171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.614135    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24415d0-ef79-46c2-bb7f-4f44d44fea97-kube-api-access-lznsc\" (UniqueName: \"kubernetes.io/projected/e24415d0-ef79-46c2-bb7f-4f44d44fea97-kube-api-access-lznsc\") pod \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\" (UID: \"e24415d0-ef79-46c2-bb7f-4f44d44fea97\") "
Nov 03 08:15:04.614393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.614371    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24415d0-ef79-46c2-bb7f-4f44d44fea97-plugins-config-volume" pod "e24415d0-ef79-46c2-bb7f-4f44d44fea97" (UID: "e24415d0-ef79-46c2-bb7f-4f44d44fea97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:04.614939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.614916    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24415d0-ef79-46c2-bb7f-4f44d44fea97-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:04.616616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.616592    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24415d0-ef79-46c2-bb7f-4f44d44fea97-kube-api-access-lznsc" pod "e24415d0-ef79-46c2-bb7f-4f44d44fea97" (UID: "e24415d0-ef79-46c2-bb7f-4f44d44fea97"). InnerVolumeSpecName "kube-api-access-lznsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:04.716853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.716826    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznsc\" (UniqueName: \"kubernetes.io/projected/e24415d0-ef79-46c2-bb7f-4f44d44fea97-kube-api-access-lznsc\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:04.718822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.718795    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:04.871357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.871320    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" event={"ID":"3d7fad21-fe12-4d47-911d-c036b42ea3ad","Type":"ContainerStarted","Data":"4fcc65bc1fa6b8b7be2b86afc3ce6299085a82c903145a72262a2dd663503f4b"}
Nov 03 08:15:04.879116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.879092    6430 generic.go:356] "Generic (PLEG): container finished" podID="e24415d0-ef79-46c2-bb7f-4f44d44fea97" containerID="fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a" exitCode=0
Nov 03 08:15:04.880342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.879289    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"
Nov 03 08:15:04.880342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.879300    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz" event={"ID":"e24415d0-ef79-46c2-bb7f-4f44d44fea97","Type":"ContainerDied","Data":"fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a"}
Nov 03 08:15:04.880342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.879340    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz" event={"ID":"e24415d0-ef79-46c2-bb7f-4f44d44fea97","Type":"ContainerDied","Data":"87219626ee67dc32c42032de602e16388c79f8aef1ea3c8d75c8f487b9637ebd"}
Nov 03 08:15:04.880342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.879365    6430 scope.go:122] "RemoveContainer" containerID="fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a"
Nov 03 08:15:04.883423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.882800    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" event={"ID":"7f06d813-d72f-4516-90b1-55419f11af20","Type":"ContainerStarted","Data":"0f9dd405ee4160b1a68a49e04c6ebd6f42e916720c12881810e14c1495b624f3"}
Nov 03 08:15:04.885269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.885243    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" event={"ID":"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5","Type":"ContainerStarted","Data":"74a7e318e91f6dfd7fdf82835b70bea45279f5a785d0a988edcce4944ab1c3e7"}
Nov 03 08:15:04.887349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.887326    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" event={"ID":"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e","Type":"ContainerStarted","Data":"eb844265424d9c368405d66a067d2a122752460e1c8383e88bba893233d09895"}
Nov 03 08:15:04.890917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.890880    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerStarted","Data":"86d1e71ab698df2141b7184848152b1e9a99835e798eb964e87051b6fe76f0a9"}
Nov 03 08:15:04.893715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.893697    6430 scope.go:122] "RemoveContainer" containerID="fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a"
Nov 03 08:15:04.894022 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:04.894001    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a\": container with ID starting with fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a not found: ID does not exist" containerID="fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a"
Nov 03 08:15:04.894101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.894031    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a"} err="failed to get container status \"fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a\": rpc error: code = NotFound desc = could not find container \"fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a\": container with ID starting with fdb74ddd4ecba05169814ee2873e68481bb4f267fd49dc916c338a10bb5d7e8a not found: ID does not exist"
Nov 03 08:15:04.907923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.907877    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" podStartSLOduration=4.907864157 podStartE2EDuration="4.907864157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:04.901871878 +0000 UTC m=+1254.404707393" watchObservedRunningTime="2026-11-03 08:15:04.907864157 +0000 UTC m=+1254.410699674"
Nov 03 08:15:04.938009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.937985    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"]
Nov 03 08:15:04.943724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.943700    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94e3cb-5854c68d4c-47blz"]
Nov 03 08:15:04.959426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:04.959379    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" podStartSLOduration=4.95936318 podStartE2EDuration="4.95936318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:04.954861187 +0000 UTC m=+1254.457696701" watchObservedRunningTime="2026-11-03 08:15:04.95936318 +0000 UTC m=+1254.462198698"
Nov 03 08:15:05.576760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.576728    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:05.718518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.718463    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:05.901592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.901505    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerID="4fcc65bc1fa6b8b7be2b86afc3ce6299085a82c903145a72262a2dd663503f4b" exitCode=60
Nov 03 08:15:05.901592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.901574    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" event={"ID":"3d7fad21-fe12-4d47-911d-c036b42ea3ad","Type":"ContainerDied","Data":"4fcc65bc1fa6b8b7be2b86afc3ce6299085a82c903145a72262a2dd663503f4b"}
Nov 03 08:15:05.901823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.901668    6430 scope.go:122] "RemoveContainer" containerID="0a1a4fef2f84b57ed7de0c0f719fa4575a6ff7ca94b2a09c613799dcf12d4538"
Nov 03 08:15:05.902162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.902134    6430 scope.go:122] "RemoveContainer" containerID="4fcc65bc1fa6b8b7be2b86afc3ce6299085a82c903145a72262a2dd663503f4b"
Nov 03 08:15:05.902589 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:05.902567    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs_odh-ai-gateway-infra(3d7fad21-fe12-4d47-911d-c036b42ea3ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad"
Nov 03 08:15:05.905392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.905373    6430 generic.go:356] "Generic (PLEG): container finished" podID="7f06d813-d72f-4516-90b1-55419f11af20" containerID="0f9dd405ee4160b1a68a49e04c6ebd6f42e916720c12881810e14c1495b624f3" exitCode=60
Nov 03 08:15:05.905474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.905450    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" event={"ID":"7f06d813-d72f-4516-90b1-55419f11af20","Type":"ContainerDied","Data":"0f9dd405ee4160b1a68a49e04c6ebd6f42e916720c12881810e14c1495b624f3"}
Nov 03 08:15:05.905772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.905752    6430 scope.go:122] "RemoveContainer" containerID="0f9dd405ee4160b1a68a49e04c6ebd6f42e916720c12881810e14c1495b624f3"
Nov 03 08:15:05.906136 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:05.906113    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2_odh-ai-gateway-infra(7f06d813-d72f-4516-90b1-55419f11af20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" podUID="7f06d813-d72f-4516-90b1-55419f11af20"
Nov 03 08:15:05.913278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.913253    6430 generic.go:356] "Generic (PLEG): container finished" podID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerID="74a7e318e91f6dfd7fdf82835b70bea45279f5a785d0a988edcce4944ab1c3e7" exitCode=60
Nov 03 08:15:05.913407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.913333    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" event={"ID":"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5","Type":"ContainerDied","Data":"74a7e318e91f6dfd7fdf82835b70bea45279f5a785d0a988edcce4944ab1c3e7"}
Nov 03 08:15:05.913739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.913720    6430 scope.go:122] "RemoveContainer" containerID="74a7e318e91f6dfd7fdf82835b70bea45279f5a785d0a988edcce4944ab1c3e7"
Nov 03 08:15:05.914048 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:05.914026    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth_odh-ai-gateway-infra(fb9e252c-56a1-483a-af31-b5a7eb2d1ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5"
Nov 03 08:15:05.915912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.915893    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerID="eb844265424d9c368405d66a067d2a122752460e1c8383e88bba893233d09895" exitCode=60
Nov 03 08:15:05.916215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.915986    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" event={"ID":"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e","Type":"ContainerDied","Data":"eb844265424d9c368405d66a067d2a122752460e1c8383e88bba893233d09895"}
Nov 03 08:15:05.916344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.916260    6430 scope.go:122] "RemoveContainer" containerID="eb844265424d9c368405d66a067d2a122752460e1c8383e88bba893233d09895"
Nov 03 08:15:05.916739 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:05.916714    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t_odh-ai-gateway-infra(3d812ab2-d05b-4bc6-b8d1-2cf70390c34e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e"
Nov 03 08:15:05.918451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.918435    6430 scope.go:122] "RemoveContainer" containerID="7a4739992498f95d13b32ece20b498cd5b81287cea294dc9a1458fb92480a46a"
Nov 03 08:15:05.919309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.919287    6430 generic.go:356] "Generic (PLEG): container finished" podID="5714d107-1408-451e-be96-16017082f797" containerID="86d1e71ab698df2141b7184848152b1e9a99835e798eb964e87051b6fe76f0a9" exitCode=60
Nov 03 08:15:05.919395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.919370    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerDied","Data":"86d1e71ab698df2141b7184848152b1e9a99835e798eb964e87051b6fe76f0a9"}
Nov 03 08:15:05.919575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.919548    6430 scope.go:122] "RemoveContainer" containerID="86d1e71ab698df2141b7184848152b1e9a99835e798eb964e87051b6fe76f0a9"
Nov 03 08:15:05.919828 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:05.919808    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g_odh-ai-gateway-infra(5714d107-1408-451e-be96-16017082f797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" podUID="5714d107-1408-451e-be96-16017082f797"
Nov 03 08:15:05.921912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.921894    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:05.932514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.932497    6430 scope.go:122] "RemoveContainer" containerID="fd6fc3f9026e8faa66a3b56d579e8ef847b44e3fc32b061b61435ba7d1a27390"
Nov 03 08:15:05.946432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.946414    6430 scope.go:122] "RemoveContainer" containerID="5ae1fbf650fd55f5d433f6c316194ab27ac1ccb1991a0d10d80a045fed87cb20"
Nov 03 08:15:05.959925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:05.959906    6430 scope.go:122] "RemoveContainer" containerID="a0c9177eb35ac64881a9dc9a6813ddb02d9ae154f3e8fd722eb06e91a0f398ed"
Nov 03 08:15:06.620871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.620839    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24415d0-ef79-46c2-bb7f-4f44d44fea97" path="/var/lib/kubelet/pods/e24415d0-ef79-46c2-bb7f-4f44d44fea97/volumes"
Nov 03 08:15:06.680608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.680567    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:06.816436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.816357    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:06.929447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.929361    6430 scope.go:122] "RemoveContainer" containerID="eb844265424d9c368405d66a067d2a122752460e1c8383e88bba893233d09895"
Nov 03 08:15:06.929829 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:06.929804    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t_odh-ai-gateway-infra(3d812ab2-d05b-4bc6-b8d1-2cf70390c34e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e"
Nov 03 08:15:06.931016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.930994    6430 scope.go:122] "RemoveContainer" containerID="86d1e71ab698df2141b7184848152b1e9a99835e798eb964e87051b6fe76f0a9"
Nov 03 08:15:06.931372 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:06.931349    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g_odh-ai-gateway-infra(5714d107-1408-451e-be96-16017082f797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" podUID="5714d107-1408-451e-be96-16017082f797"
Nov 03 08:15:06.932577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.932562    6430 scope.go:122] "RemoveContainer" containerID="4fcc65bc1fa6b8b7be2b86afc3ce6299085a82c903145a72262a2dd663503f4b"
Nov 03 08:15:06.932886 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:06.932863    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs_odh-ai-gateway-infra(3d7fad21-fe12-4d47-911d-c036b42ea3ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad"
Nov 03 08:15:06.934129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.934111    6430 scope.go:122] "RemoveContainer" containerID="0f9dd405ee4160b1a68a49e04c6ebd6f42e916720c12881810e14c1495b624f3"
Nov 03 08:15:06.934387 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:06.934365    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2_odh-ai-gateway-infra(7f06d813-d72f-4516-90b1-55419f11af20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" podUID="7f06d813-d72f-4516-90b1-55419f11af20"
Nov 03 08:15:06.935712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:06.935697    6430 scope.go:122] "RemoveContainer" containerID="74a7e318e91f6dfd7fdf82835b70bea45279f5a785d0a988edcce4944ab1c3e7"
Nov 03 08:15:06.935958 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:06.935940    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth_odh-ai-gateway-infra(fb9e252c-56a1-483a-af31-b5a7eb2d1ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5"
Nov 03 08:15:07.122086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.115529    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:07.608090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.607914    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:07.660506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.660469    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"]
Nov 03 08:15:07.661773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.661743    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e24415d0-ef79-46c2-bb7f-4f44d44fea97" containerName="payload-pre-processing"
Nov 03 08:15:07.661773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.661774    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24415d0-ef79-46c2-bb7f-4f44d44fea97" containerName="payload-pre-processing"
Nov 03 08:15:07.661961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.661947    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24415d0-ef79-46c2-bb7f-4f44d44fea97" containerName="payload-pre-processing"
Nov 03 08:15:07.669572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.669535    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"]
Nov 03 08:15:07.669715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.669650    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:07.673534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.673510    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-179c16\""
Nov 03 08:15:07.673662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.673650    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-179c16-dockercfg-mslsr\""
Nov 03 08:15:07.748596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.748508    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:07.825095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.824386    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdv9d\" (UniqueName: \"kubernetes.io/projected/475aa1d3-6841-4350-959b-85d2ac5517bf-kube-api-access-rdv9d\") pod \"payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:07.825095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.824528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475aa1d3-6841-4350-959b-85d2ac5517bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:07.932179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.931594    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475aa1d3-6841-4350-959b-85d2ac5517bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:07.932179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.931944    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdv9d\" (UniqueName: \"kubernetes.io/projected/475aa1d3-6841-4350-959b-85d2ac5517bf-kube-api-access-rdv9d\") pod \"payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:07.944155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:07.933359    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475aa1d3-6841-4350-959b-85d2ac5517bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:08.034341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:08.028133    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdv9d\" (UniqueName: \"kubernetes.io/projected/475aa1d3-6841-4350-959b-85d2ac5517bf-kube-api-access-rdv9d\") pod \"payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:08.312114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:08.311980    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:08.355846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:08.355811    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:08.708943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:08.708914    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"]
Nov 03 08:15:08.710179 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:08.710151    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475aa1d3_6841_4350_959b_85d2ac5517bf.slice/crio-b78474a056b4ba7a328441a3337e5d96ab50864df8c4154875de45b7174ff084 WatchSource:0}: Error finding container b78474a056b4ba7a328441a3337e5d96ab50864df8c4154875de45b7174ff084: Status 404 returned error can't find the container with id b78474a056b4ba7a328441a3337e5d96ab50864df8c4154875de45b7174ff084
Nov 03 08:15:09.020402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:09.020296    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" event={"ID":"475aa1d3-6841-4350-959b-85d2ac5517bf","Type":"ContainerStarted","Data":"843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd"}
Nov 03 08:15:09.020402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:09.020368    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" event={"ID":"475aa1d3-6841-4350-959b-85d2ac5517bf","Type":"ContainerStarted","Data":"b78474a056b4ba7a328441a3337e5d96ab50864df8c4154875de45b7174ff084"}
Nov 03 08:15:09.020402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:09.020402    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:09.042122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:09.042046    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" podStartSLOduration=2.042028085 podStartE2EDuration="2.042028085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:09.038440038 +0000 UTC m=+1258.541275552" watchObservedRunningTime="2026-11-03 08:15:09.042028085 +0000 UTC m=+1258.544863600"
Nov 03 08:15:09.294135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:09.287964    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:09.390927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:09.390884    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:10.284666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:10.284631    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:10.413086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:10.410130    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:10.619022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:10.618943    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:11.495573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:11.495537    6430 scope.go:122] "RemoveContainer" containerID="e1adbfb90d116b9caeadb415c4ebf106e1b9e8e56378e6c07673079747124ab6"
Nov 03 08:15:11.588408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:11.588362    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:12.393454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:12.393421    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:12.649700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:12.649615    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:13.415277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:13.415221    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:14.341936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:14.327520    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:14.894239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:14.894213    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:15:15.447189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:15.447158    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:15.614087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:15.613916    6430 scope.go:122] "RemoveContainer" containerID="86d1e71ab698df2141b7184848152b1e9a99835e798eb964e87051b6fe76f0a9"
Nov 03 08:15:16.103825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.103790    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerStarted","Data":"2d28070f0c1edee355b7529417175d743bce5933b86daf16c00c7517dc12226a"}
Nov 03 08:15:16.342927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.342851    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:16.479511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.479478    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:16.556091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.556039    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"]
Nov 03 08:15:16.556544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.556508    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy" containerID="cri-o://bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17" gracePeriod=30
Nov 03 08:15:16.638962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.635442    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:16.642161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.642137    6430 scope.go:122] "RemoveContainer" containerID="74a7e318e91f6dfd7fdf82835b70bea45279f5a785d0a988edcce4944ab1c3e7"
Nov 03 08:15:16.800305 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:16.799978    6430 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5714d107_1408_451e_be96_16017082f797.slice/crio-2d28070f0c1edee355b7529417175d743bce5933b86daf16c00c7517dc12226a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5714d107_1408_451e_be96_16017082f797.slice/crio-conmon-2d28070f0c1edee355b7529417175d743bce5933b86daf16c00c7517dc12226a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:15:16.800305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.800096    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"]
Nov 03 08:15:16.800522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.800338    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy" containerID="cri-o://fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420" gracePeriod=30
Nov 03 08:15:16.853953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:16.853900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:17.043313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.043281    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"]
Nov 03 08:15:17.043539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.043514    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy" containerID="cri-o://9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34" gracePeriod=30
Nov 03 08:15:17.114537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.114509    6430 generic.go:356] "Generic (PLEG): container finished" podID="5714d107-1408-451e-be96-16017082f797" containerID="2d28070f0c1edee355b7529417175d743bce5933b86daf16c00c7517dc12226a" exitCode=60
Nov 03 08:15:17.114720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.114583    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerDied","Data":"2d28070f0c1edee355b7529417175d743bce5933b86daf16c00c7517dc12226a"}
Nov 03 08:15:17.114720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.114655    6430 scope.go:122] "RemoveContainer" containerID="86d1e71ab698df2141b7184848152b1e9a99835e798eb964e87051b6fe76f0a9"
Nov 03 08:15:17.115182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.115164    6430 scope.go:122] "RemoveContainer" containerID="2d28070f0c1edee355b7529417175d743bce5933b86daf16c00c7517dc12226a"
Nov 03 08:15:17.115623 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:17.115598    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g_odh-ai-gateway-infra(5714d107-1408-451e-be96-16017082f797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" podUID="5714d107-1408-451e-be96-16017082f797"
Nov 03 08:15:17.158114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.157882    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"]
Nov 03 08:15:17.160231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.160185    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:17.615146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.612043    6430 scope.go:122] "RemoveContainer" containerID="4fcc65bc1fa6b8b7be2b86afc3ce6299085a82c903145a72262a2dd663503f4b"
Nov 03 08:15:17.762095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:17.759719    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:18.173134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.152324    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:18.190901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.190869    6430 generic.go:356] "Generic (PLEG): container finished" podID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerID="7cc00f9a516fcc42933259af4d491bae735a441e5aac20181be44935c2f7eae8" exitCode=60
Nov 03 08:15:18.191369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.191344    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" event={"ID":"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5","Type":"ContainerDied","Data":"7cc00f9a516fcc42933259af4d491bae735a441e5aac20181be44935c2f7eae8"}
Nov 03 08:15:18.191567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.191542    6430 scope.go:122] "RemoveContainer" containerID="74a7e318e91f6dfd7fdf82835b70bea45279f5a785d0a988edcce4944ab1c3e7"
Nov 03 08:15:18.192262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.192244    6430 scope.go:122] "RemoveContainer" containerID="7cc00f9a516fcc42933259af4d491bae735a441e5aac20181be44935c2f7eae8"
Nov 03 08:15:18.255287 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:18.255200    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth_odh-ai-gateway-infra(fb9e252c-56a1-483a-af31-b5a7eb2d1ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5"
Nov 03 08:15:18.293723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.293695    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:18.432089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.426927    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"]
Nov 03 08:15:18.445110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.440929    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.460085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.456827    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.460085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.456976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.460085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.457092    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgx85\" (UniqueName: \"kubernetes.io/projected/4034aaa5-c9b9-4b51-b163-981403892ed6-kube-api-access-bgx85\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.469096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.463675    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"]
Nov 03 08:15:18.481776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.480581    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:15:18.559920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.559833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.560113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.559933    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.560113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.560023    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgx85\" (UniqueName: \"kubernetes.io/projected/4034aaa5-c9b9-4b51-b163-981403892ed6-kube-api-access-bgx85\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.561363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.561220    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.569744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.569697    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgx85\" (UniqueName: \"kubernetes.io/projected/4034aaa5-c9b9-4b51-b163-981403892ed6-kube-api-access-bgx85\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.570474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.570426    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bae2c8-5l7pd\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.626382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.625877    6430 scope.go:122] "RemoveContainer" containerID="0f9dd405ee4160b1a68a49e04c6ebd6f42e916720c12881810e14c1495b624f3"
Nov 03 08:15:18.629944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.628970    6430 scope.go:122] "RemoveContainer" containerID="eb844265424d9c368405d66a067d2a122752460e1c8383e88bba893233d09895"
Nov 03 08:15:18.729520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.729495    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:18.763415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.763372    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5714d107-1408-451e-be96-16017082f797-kube-api-access-mmfxs\" (UniqueName: \"kubernetes.io/projected/5714d107-1408-451e-be96-16017082f797-kube-api-access-mmfxs\") pod \"5714d107-1408-451e-be96-16017082f797\" (UID: \"5714d107-1408-451e-be96-16017082f797\") "
Nov 03 08:15:18.763729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.763706    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5714d107-1408-451e-be96-16017082f797-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714d107-1408-451e-be96-16017082f797-maas-api-service-ca\") pod \"5714d107-1408-451e-be96-16017082f797\" (UID: \"5714d107-1408-451e-be96-16017082f797\") "
Nov 03 08:15:18.764146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.764104    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5714d107-1408-451e-be96-16017082f797-maas-api-service-ca" pod "5714d107-1408-451e-be96-16017082f797" (UID: "5714d107-1408-451e-be96-16017082f797"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:18.765100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.765077    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714d107-1408-451e-be96-16017082f797-maas-api-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:18.767805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.767773    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5714d107-1408-451e-be96-16017082f797-kube-api-access-mmfxs" pod "5714d107-1408-451e-be96-16017082f797" (UID: "5714d107-1408-451e-be96-16017082f797"). InnerVolumeSpecName "kube-api-access-mmfxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:18.838933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.838844    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:18.867797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.867764    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfxs\" (UniqueName: \"kubernetes.io/projected/5714d107-1408-451e-be96-16017082f797-kube-api-access-mmfxs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:18.953151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:18.952742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:19.189685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.189565    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"]
Nov 03 08:15:19.190412 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:19.190386    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4034aaa5_c9b9_4b51_b163_981403892ed6.slice/crio-18b2aad5b4519c3f9fcb27c282fae5b0230ae07fd9d2878f311f7b697aab7754 WatchSource:0}: Error finding container 18b2aad5b4519c3f9fcb27c282fae5b0230ae07fd9d2878f311f7b697aab7754: Status 404 returned error can't find the container with id 18b2aad5b4519c3f9fcb27c282fae5b0230ae07fd9d2878f311f7b697aab7754
Nov 03 08:15:19.279316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.279283    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"]
Nov 03 08:15:19.291039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.291008    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd" event={"ID":"4034aaa5-c9b9-4b51-b163-981403892ed6","Type":"ContainerStarted","Data":"18b2aad5b4519c3f9fcb27c282fae5b0230ae07fd9d2878f311f7b697aab7754"}
Nov 03 08:15:19.292880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.292854    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g" event={"ID":"5714d107-1408-451e-be96-16017082f797","Type":"ContainerDied","Data":"3b66f91cf817fe303cd38a66d5a12cceb87782aa4738d29d37b963342f94f24d"}
Nov 03 08:15:19.293007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.292931    6430 scope.go:122] "RemoveContainer" containerID="2d28070f0c1edee355b7529417175d743bce5933b86daf16c00c7517dc12226a"
Nov 03 08:15:19.293536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.293506    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"
Nov 03 08:15:19.302114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.302037    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerID="a0fd7acce7f9b6bad98dcd28f5bc185f72961dea2e31dd45725e00267078def7" exitCode=60
Nov 03 08:15:19.302216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.302138    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" event={"ID":"3d7fad21-fe12-4d47-911d-c036b42ea3ad","Type":"ContainerDied","Data":"a0fd7acce7f9b6bad98dcd28f5bc185f72961dea2e31dd45725e00267078def7"}
Nov 03 08:15:19.302918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.302887    6430 scope.go:122] "RemoveContainer" containerID="a0fd7acce7f9b6bad98dcd28f5bc185f72961dea2e31dd45725e00267078def7"
Nov 03 08:15:19.303421 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:19.303378    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs_odh-ai-gateway-infra(3d7fad21-fe12-4d47-911d-c036b42ea3ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad"
Nov 03 08:15:19.310669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.310642    6430 scope.go:122] "RemoveContainer" containerID="4fcc65bc1fa6b8b7be2b86afc3ce6299085a82c903145a72262a2dd663503f4b"
Nov 03 08:15:19.313649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.313627    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" event={"ID":"7f06d813-d72f-4516-90b1-55419f11af20","Type":"ContainerStarted","Data":"c1b795320c966daf66e81c1c68ea5229a9bad91650213dc577e4d31d7ae81386"}
Nov 03 08:15:19.338676 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.338654    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"]
Nov 03 08:15:19.342596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.342545    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6bc553-29894895-fmg6g"]
Nov 03 08:15:19.580976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.580953    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:19.682825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.682796    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-maas-api-service-ca\") pod \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") "
Nov 03 08:15:19.683197 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.683011    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-kube-api-access-sbz87\" (UniqueName: \"kubernetes.io/projected/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-kube-api-access-sbz87\") pod \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\" (UID: \"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5\") "
Nov 03 08:15:19.683702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.683654    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-maas-api-service-ca" pod "fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" (UID: "fb9e252c-56a1-483a-af31-b5a7eb2d1ff5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:19.686129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.686045    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-kube-api-access-sbz87" pod "fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" (UID: "fb9e252c-56a1-483a-af31-b5a7eb2d1ff5"). InnerVolumeSpecName "kube-api-access-sbz87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:19.786136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.786104    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-maas-api-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:19.786136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.786137    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbz87\" (UniqueName: \"kubernetes.io/projected/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5-kube-api-access-sbz87\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:19.795221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.795192    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:19.925203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:19.916159    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:20.090109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.088265    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:15:20.126101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.123169    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:20.322802 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.322717    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth" event={"ID":"fb9e252c-56a1-483a-af31-b5a7eb2d1ff5","Type":"ContainerDied","Data":"148876e80b0d4096f11b2f9c3fda91b60a39e680cde70cb8a2e56eada5a10e2d"}
Nov 03 08:15:20.322802 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.322743    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"
Nov 03 08:15:20.322802 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.322795    6430 scope.go:122] "RemoveContainer" containerID="7cc00f9a516fcc42933259af4d491bae735a441e5aac20181be44935c2f7eae8"
Nov 03 08:15:20.325125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.325097    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd" event={"ID":"4034aaa5-c9b9-4b51-b163-981403892ed6","Type":"ContainerStarted","Data":"7f0f92c51a6ecc7bc24058cca14aafc8f646684010fae68503fad33c98216bbd"}
Nov 03 08:15:20.331012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.329433    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerID="a34c14ea1beff17cdd75081a52d8e967342da5f433e2becccf9e4b753c6e64bb" exitCode=60
Nov 03 08:15:20.331012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.329522    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" event={"ID":"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e","Type":"ContainerDied","Data":"a34c14ea1beff17cdd75081a52d8e967342da5f433e2becccf9e4b753c6e64bb"}
Nov 03 08:15:20.331012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.329886    6430 scope.go:122] "RemoveContainer" containerID="a34c14ea1beff17cdd75081a52d8e967342da5f433e2becccf9e4b753c6e64bb"
Nov 03 08:15:20.331012 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:20.330181    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t_odh-ai-gateway-infra(3d812ab2-d05b-4bc6-b8d1-2cf70390c34e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e"
Nov 03 08:15:20.338010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.337986    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"]
Nov 03 08:15:20.345251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.345230    6430 scope.go:122] "RemoveContainer" containerID="eb844265424d9c368405d66a067d2a122752460e1c8383e88bba893233d09895"
Nov 03 08:15:20.346860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.346836    6430 generic.go:356] "Generic (PLEG): container finished" podID="7f06d813-d72f-4516-90b1-55419f11af20" containerID="c1b795320c966daf66e81c1c68ea5229a9bad91650213dc577e4d31d7ae81386" exitCode=60
Nov 03 08:15:20.346952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.346908    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" event={"ID":"7f06d813-d72f-4516-90b1-55419f11af20","Type":"ContainerDied","Data":"c1b795320c966daf66e81c1c68ea5229a9bad91650213dc577e4d31d7ae81386"}
Nov 03 08:15:20.348584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.347533    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd" podStartSLOduration=2.347515957 podStartE2EDuration="2.347515957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:20.345589993 +0000 UTC m=+1269.848425508" watchObservedRunningTime="2026-11-03 08:15:20.347515957 +0000 UTC m=+1269.850351520"
Nov 03 08:15:20.353304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.352367    6430 scope.go:122] "RemoveContainer" containerID="c1b795320c966daf66e81c1c68ea5229a9bad91650213dc577e4d31d7ae81386"
Nov 03 08:15:20.355149 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:20.354194    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2_odh-ai-gateway-infra(7f06d813-d72f-4516-90b1-55419f11af20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" podUID="7f06d813-d72f-4516-90b1-55419f11af20"
Nov 03 08:15:20.642392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.642332    6430 scope.go:122] "RemoveContainer" containerID="0f9dd405ee4160b1a68a49e04c6ebd6f42e916720c12881810e14c1495b624f3"
Nov 03 08:15:20.824582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.824554    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5714d107-1408-451e-be96-16017082f797" path="/var/lib/kubelet/pods/5714d107-1408-451e-be96-16017082f797/volumes"
Nov 03 08:15:20.825453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.825432    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"]
Nov 03 08:15:20.833945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.833910    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"]
Nov 03 08:15:20.838623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.838600    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a61a48-29894895-dnxth"]
Nov 03 08:15:20.910935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.910837    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:20.966656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:20.966627    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:21.160110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.149261    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7fad21-fe12-4d47-911d-c036b42ea3ad-kube-api-access-pvngh\" (UniqueName: \"kubernetes.io/projected/3d7fad21-fe12-4d47-911d-c036b42ea3ad-kube-api-access-pvngh\") pod \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") "
Nov 03 08:15:21.160110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.149610    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7fad21-fe12-4d47-911d-c036b42ea3ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7fad21-fe12-4d47-911d-c036b42ea3ad-maas-api-service-ca\") pod \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\" (UID: \"3d7fad21-fe12-4d47-911d-c036b42ea3ad\") "
Nov 03 08:15:21.160110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.151217    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7fad21-fe12-4d47-911d-c036b42ea3ad-maas-api-service-ca" pod "3d7fad21-fe12-4d47-911d-c036b42ea3ad" (UID: "3d7fad21-fe12-4d47-911d-c036b42ea3ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:21.170496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.170408    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7fad21-fe12-4d47-911d-c036b42ea3ad-kube-api-access-pvngh" pod "3d7fad21-fe12-4d47-911d-c036b42ea3ad" (UID: "3d7fad21-fe12-4d47-911d-c036b42ea3ad"). InnerVolumeSpecName "kube-api-access-pvngh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:21.251998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.251967    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7fad21-fe12-4d47-911d-c036b42ea3ad-maas-api-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.251998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.251994    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvngh\" (UniqueName: \"kubernetes.io/projected/3d7fad21-fe12-4d47-911d-c036b42ea3ad-kube-api-access-pvngh\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.355614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.355583    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"
Nov 03 08:15:21.355814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.355587    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs" event={"ID":"3d7fad21-fe12-4d47-911d-c036b42ea3ad","Type":"ContainerDied","Data":"08513d7b8645af5a014f6fcba14c39ff1c8b349cafe5d61a0601fe43dd0e0e55"}
Nov 03 08:15:21.355814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.355726    6430 scope.go:122] "RemoveContainer" containerID="a0fd7acce7f9b6bad98dcd28f5bc185f72961dea2e31dd45725e00267078def7"
Nov 03 08:15:21.367923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.367894    6430 generic.go:356] "Generic (PLEG): container finished" podID="4034aaa5-c9b9-4b51-b163-981403892ed6" containerID="7f0f92c51a6ecc7bc24058cca14aafc8f646684010fae68503fad33c98216bbd" exitCode=0
Nov 03 08:15:21.368046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.367973    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd" event={"ID":"4034aaa5-c9b9-4b51-b163-981403892ed6","Type":"ContainerDied","Data":"7f0f92c51a6ecc7bc24058cca14aafc8f646684010fae68503fad33c98216bbd"}
Nov 03 08:15:21.386947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.386912    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"]
Nov 03 08:15:21.397948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.397914    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"]
Nov 03 08:15:21.403629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.403603    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-950e06-29894895-nz4fs"]
Nov 03 08:15:21.599944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.599921    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:21.661756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.661731    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:21.760833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.760799    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-kube-api-access-n4wsz\" (UniqueName: \"kubernetes.io/projected/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-kube-api-access-n4wsz\") pod \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") "
Nov 03 08:15:21.760980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.760863    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f06d813-d72f-4516-90b1-55419f11af20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f06d813-d72f-4516-90b1-55419f11af20-maas-api-service-ca\") pod \"7f06d813-d72f-4516-90b1-55419f11af20\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") "
Nov 03 08:15:21.761098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.761057    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f06d813-d72f-4516-90b1-55419f11af20-kube-api-access-qzbsr\" (UniqueName: \"kubernetes.io/projected/7f06d813-d72f-4516-90b1-55419f11af20-kube-api-access-qzbsr\") pod \"7f06d813-d72f-4516-90b1-55419f11af20\" (UID: \"7f06d813-d72f-4516-90b1-55419f11af20\") "
Nov 03 08:15:21.761156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.761107    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-maas-api-service-ca\") pod \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\" (UID: \"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e\") "
Nov 03 08:15:21.761512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.761490    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-maas-api-service-ca" pod "3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" (UID: "3d812ab2-d05b-4bc6-b8d1-2cf70390c34e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:21.761547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.761528    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f06d813-d72f-4516-90b1-55419f11af20-maas-api-service-ca" pod "7f06d813-d72f-4516-90b1-55419f11af20" (UID: "7f06d813-d72f-4516-90b1-55419f11af20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:21.762041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.762009    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f06d813-d72f-4516-90b1-55419f11af20-maas-api-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.762041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.762037    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-maas-api-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.763478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.763453    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f06d813-d72f-4516-90b1-55419f11af20-kube-api-access-qzbsr" pod "7f06d813-d72f-4516-90b1-55419f11af20" (UID: "7f06d813-d72f-4516-90b1-55419f11af20"). InnerVolumeSpecName "kube-api-access-qzbsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:21.763771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.763751    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-kube-api-access-n4wsz" pod "3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" (UID: "3d812ab2-d05b-4bc6-b8d1-2cf70390c34e"). InnerVolumeSpecName "kube-api-access-n4wsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:21.864096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.864045    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzbsr\" (UniqueName: \"kubernetes.io/projected/7f06d813-d72f-4516-90b1-55419f11af20-kube-api-access-qzbsr\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.864096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.864098    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4wsz\" (UniqueName: \"kubernetes.io/projected/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e-kube-api-access-n4wsz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.917671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:21.917639    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090253    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-data\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090315    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13e73f76-be64-4daf-89ec-09cbea76457c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e73f76-be64-4daf-89ec-09cbea76457c-istio-podinfo\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090493    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-istio-token\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-istio-token\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090533    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-socket\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090567    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e73f76-be64-4daf-89ec-09cbea76457c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e73f76-be64-4daf-89ec-09cbea76457c-istiod-ca-cert\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090633    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-envoy\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090746    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-kube-api-access-n57b5\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-kube-api-access-n57b5\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090823    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-credential-socket\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.092090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.090873    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-certs\") pod \"13e73f76-be64-4daf-89ec-09cbea76457c\" (UID: \"13e73f76-be64-4daf-89ec-09cbea76457c\") "
Nov 03 08:15:22.093329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.093295    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-certs" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.105806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.101403    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-socket" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.105806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.102257    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-data" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.114128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.113928    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e73f76-be64-4daf-89ec-09cbea76457c-istiod-ca-cert" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:22.129087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.126118    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13e73f76-be64-4daf-89ec-09cbea76457c-istio-podinfo" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:15:22.129087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.126336    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-istio-token" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.129513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.129436    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-credential-socket" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.139706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.139620    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-envoy" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.146405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.146342    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-kube-api-access-n57b5" pod "13e73f76-be64-4daf-89ec-09cbea76457c" (UID: "13e73f76-be64-4daf-89ec-09cbea76457c"). InnerVolumeSpecName "kube-api-access-n57b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.165241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.165139    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195165    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195196    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e73f76-be64-4daf-89ec-09cbea76457c-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195212    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195224    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195238    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e73f76-be64-4daf-89ec-09cbea76457c-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195251    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195265    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n57b5\" (UniqueName: \"kubernetes.io/projected/13e73f76-be64-4daf-89ec-09cbea76457c-kube-api-access-n57b5\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195280    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.195315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.195293    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e73f76-be64-4daf-89ec-09cbea76457c-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.230437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.230409    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:15:22.288186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.288106    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:22.357813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.357790    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:15:22.392678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.392608    6430 generic.go:356] "Generic (PLEG): container finished" podID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerID="9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34" exitCode=0
Nov 03 08:15:22.393232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.393002    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"
Nov 03 08:15:22.393767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.393022    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" event={"ID":"46bea25b-da8a-4336-aeaf-f82b3e59d563","Type":"ContainerDied","Data":"9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34"}
Nov 03 08:15:22.393943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.393923    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn" event={"ID":"46bea25b-da8a-4336-aeaf-f82b3e59d563","Type":"ContainerDied","Data":"1b914ba5b2dcce2815654e61723f0a1bfd4fbd9e4d98b924d80ee79c2670fbd6"}
Nov 03 08:15:22.394082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.394052    6430 scope.go:122] "RemoveContainer" containerID="9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34"
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397316    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-kube-api-access-fl2cq\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-kube-api-access-fl2cq\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397383    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-podinfo\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397427    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-credential-socket\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397466    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-data\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397598    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-token\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-token\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397644    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-envoy\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397821    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5231c00e-b16d-40a8-9dbe-76f1509d758b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5231c00e-b16d-40a8-9dbe-76f1509d758b-istiod-ca-cert\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.398180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.397943    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-certs\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.399747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.398736    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-socket\") pod \"5231c00e-b16d-40a8-9dbe-76f1509d758b\" (UID: \"5231c00e-b16d-40a8-9dbe-76f1509d758b\") "
Nov 03 08:15:22.411085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.399893    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-socket" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.438691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.434488    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-credential-socket" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.438691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.434857    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-certs" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.438691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.435335    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-data" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.506084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.460243    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5231c00e-b16d-40a8-9dbe-76f1509d758b-istiod-ca-cert" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:22.506084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.477681    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.509850    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-data\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.509927    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-token\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-token\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.509978    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-certs\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.510108    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-kube-api-access-72sxx\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-kube-api-access-72sxx\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.510185    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-podinfo\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.510279    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bea25b-da8a-4336-aeaf-f82b3e59d563-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bea25b-da8a-4336-aeaf-f82b3e59d563-istiod-ca-cert\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.510334    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-credential-socket\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.510493    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-envoy\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.518093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.510565    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-socket\") pod \"46bea25b-da8a-4336-aeaf-f82b3e59d563\" (UID: \"46bea25b-da8a-4336-aeaf-f82b3e59d563\") "
Nov 03 08:15:22.568133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.567211    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-credential-socket" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.568133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.567963    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-data" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.568434    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2" event={"ID":"7f06d813-d72f-4516-90b1-55419f11af20","Type":"ContainerDied","Data":"59425eba77a26af4a94a9e9d3f222fba9d3703ec51c29468dca8341418ccc260"}
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.568562    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.587049    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bea25b-da8a-4336-aeaf-f82b3e59d563-istiod-ca-cert" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.588354    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-socket" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.589153    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-certs" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592035    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592082    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bea25b-da8a-4336-aeaf-f82b3e59d563-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592099    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592112    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592124    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592136    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5231c00e-b16d-40a8-9dbe-76f1509d758b-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592147    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592159    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592170    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.593096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.592182    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.609124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.609086    6430 generic.go:356] "Generic (PLEG): container finished" podID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerID="fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420" exitCode=0
Nov 03 08:15:22.626280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.626237    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" event={"ID":"5231c00e-b16d-40a8-9dbe-76f1509d758b","Type":"ContainerDied","Data":"fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420"}
Nov 03 08:15:22.626732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.626666    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt" event={"ID":"5231c00e-b16d-40a8-9dbe-76f1509d758b","Type":"ContainerDied","Data":"fecaa1f326f6a87b8f1b7d90987a59bccb6f75d65297e4d08e2a577557b76e13"}
Nov 03 08:15:22.626981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.626965    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"
Nov 03 08:15:22.655283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.639626    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"
Nov 03 08:15:22.656280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.656249    6430 scope.go:122] "RemoveContainer" containerID="9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34"
Nov 03 08:15:22.658537 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:22.658504    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34\": container with ID starting with 9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34 not found: ID does not exist" containerID="9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34"
Nov 03 08:15:22.658699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.658546    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34"} err="failed to get container status \"9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34\": rpc error: code = NotFound desc = could not find container \"9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34\": container with ID starting with 9ccfa28010ec97f0171d680a411033d3a8d071404434aa656dbea4d230d4ff34 not found: ID does not exist"
Nov 03 08:15:22.658699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.658567    6430 scope.go:122] "RemoveContainer" containerID="c1b795320c966daf66e81c1c68ea5229a9bad91650213dc577e4d31d7ae81386"
Nov 03 08:15:22.673088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.659998    6430 generic.go:356] "Generic (PLEG): container finished" podID="13e73f76-be64-4daf-89ec-09cbea76457c" containerID="bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17" exitCode=0
Nov 03 08:15:22.673088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.660400    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"
Nov 03 08:15:22.741418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.739154    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-envoy" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.744055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.740190    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-podinfo" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:15:22.768612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.768563    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-token" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.768762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.768671    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-kube-api-access-72sxx" pod "46bea25b-da8a-4336-aeaf-f82b3e59d563" (UID: "46bea25b-da8a-4336-aeaf-f82b3e59d563"). InnerVolumeSpecName "kube-api-access-72sxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.771977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.771947    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-kube-api-access-fl2cq" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "kube-api-access-fl2cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.772161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.772141    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-token" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.788077 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.787876    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" path="/var/lib/kubelet/pods/3d7fad21-fe12-4d47-911d-c036b42ea3ad/volumes"
Nov 03 08:15:22.788632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.788608    6430 scope.go:122] "RemoveContainer" containerID="fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420"
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.796477    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-podinfo" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.800861    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" path="/var/lib/kubelet/pods/fb9e252c-56a1-483a-af31-b5a7eb2d1ff5/volumes"
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.801840    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-envoy" pod "5231c00e-b16d-40a8-9dbe-76f1509d758b" (UID: "5231c00e-b16d-40a8-9dbe-76f1509d758b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.803415    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t" event={"ID":"3d812ab2-d05b-4bc6-b8d1-2cf70390c34e","Type":"ContainerDied","Data":"9f9fc1d52b77c3c4cc6ed37a995b0858396fcb3d78126b1918de4c5451b5dacc"}
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.803444    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" event={"ID":"13e73f76-be64-4daf-89ec-09cbea76457c","Type":"ContainerDied","Data":"bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17"}
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.803458    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"]
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.803489    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94e3cb-29894895-l5v9t"]
Nov 03 08:15:22.807106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.803507    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v" event={"ID":"13e73f76-be64-4daf-89ec-09cbea76457c","Type":"ContainerDied","Data":"ff6a9118a008261ba5b4dc5911f102b61a83fc64509c5899ce8bb51fbeab4963"}
Nov 03 08:15:22.809147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809120    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.809255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809161    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72sxx\" (UniqueName: \"kubernetes.io/projected/46bea25b-da8a-4336-aeaf-f82b3e59d563-kube-api-access-72sxx\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.809255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809180    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.809255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809197    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl2cq\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-kube-api-access-fl2cq\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.809255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809215    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.809255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809236    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.809255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809251    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5231c00e-b16d-40a8-9dbe-76f1509d758b-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.809433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.809266    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bea25b-da8a-4336-aeaf-f82b3e59d563-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.833868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.833836    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"]
Nov 03 08:15:22.838265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.838205    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-r2z9v"]
Nov 03 08:15:22.840320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.840253    6430 scope.go:122] "RemoveContainer" containerID="fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420"
Nov 03 08:15:22.841022 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:22.840984    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420\": container with ID starting with fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420 not found: ID does not exist" containerID="fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420"
Nov 03 08:15:22.841505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.841122    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420"} err="failed to get container status \"fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420\": rpc error: code = NotFound desc = could not find container \"fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420\": container with ID starting with fa01dc49cd17e471e6a28d650e9754dc68d4903cc6911695e1bfcc0ca6e4f420 not found: ID does not exist"
Nov 03 08:15:22.841505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.841154    6430 scope.go:122] "RemoveContainer" containerID="a34c14ea1beff17cdd75081a52d8e967342da5f433e2becccf9e4b753c6e64bb"
Nov 03 08:15:22.846887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.846867    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"]
Nov 03 08:15:22.851103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.851045    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-228695-29894895-pjkh2"]
Nov 03 08:15:22.855401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.855385    6430 scope.go:122] "RemoveContainer" containerID="bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17"
Nov 03 08:15:22.870518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.870500    6430 scope.go:122] "RemoveContainer" containerID="bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17"
Nov 03 08:15:22.873373 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:22.873306    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17\": container with ID starting with bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17 not found: ID does not exist" containerID="bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17"
Nov 03 08:15:22.873373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.873348    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17"} err="failed to get container status \"bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17\": rpc error: code = NotFound desc = could not find container \"bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17\": container with ID starting with bff702620b2c649622059abcf8945927625f44c055a6a2d968c6436dcc3c0a17 not found: ID does not exist"
Nov 03 08:15:22.900911 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:22.900873    6430 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5231c00e_b16d_40a8_9dbe_76f1509d758b.slice/crio-fecaa1f326f6a87b8f1b7d90987a59bccb6f75d65297e4d08e2a577557b76e13\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bea25b_da8a_4336_aeaf_f82b3e59d563.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bea25b_da8a_4336_aeaf_f82b3e59d563.slice/crio-1b914ba5b2dcce2815654e61723f0a1bfd4fbd9e4d98b924d80ee79c2670fbd6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f06d813_d72f_4516_90b1_55419f11af20.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5231c00e_b16d_40a8_9dbe_76f1509d758b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f06d813_d72f_4516_90b1_55419f11af20.slice/crio-59425eba77a26af4a94a9e9d3f222fba9d3703ec51c29468dca8341418ccc260\": RecentStats: unable to find data in memory cache]"
Nov 03 08:15:22.947112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.947086    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:22.949766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.949741    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"]
Nov 03 08:15:22.953755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:22.953729    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-h5vnt"]
Nov 03 08:15:23.018162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.018131    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"]
Nov 03 08:15:23.023835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.023800    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-xp2qn"]
Nov 03 08:15:23.112746 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.112666    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-openshift-ca\") pod \"4034aaa5-c9b9-4b51-b163-981403892ed6\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") "
Nov 03 08:15:23.112746 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.112706    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-xks-ca\") pod \"4034aaa5-c9b9-4b51-b163-981403892ed6\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") "
Nov 03 08:15:23.112995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.112973    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4034aaa5-c9b9-4b51-b163-981403892ed6-kube-api-access-bgx85\" (UniqueName: \"kubernetes.io/projected/4034aaa5-c9b9-4b51-b163-981403892ed6-kube-api-access-bgx85\") pod \"4034aaa5-c9b9-4b51-b163-981403892ed6\" (UID: \"4034aaa5-c9b9-4b51-b163-981403892ed6\") "
Nov 03 08:15:23.113186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.113161    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-openshift-ca" pod "4034aaa5-c9b9-4b51-b163-981403892ed6" (UID: "4034aaa5-c9b9-4b51-b163-981403892ed6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:23.113657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.113633    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.115277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.115254    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4034aaa5-c9b9-4b51-b163-981403892ed6-kube-api-access-bgx85" pod "4034aaa5-c9b9-4b51-b163-981403892ed6" (UID: "4034aaa5-c9b9-4b51-b163-981403892ed6"). InnerVolumeSpecName "kube-api-access-bgx85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:23.115490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.115464    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-xks-ca" pod "4034aaa5-c9b9-4b51-b163-981403892ed6" (UID: "4034aaa5-c9b9-4b51-b163-981403892ed6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:15:23.215272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.215239    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgx85\" (UniqueName: \"kubernetes.io/projected/4034aaa5-c9b9-4b51-b163-981403892ed6-kube-api-access-bgx85\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.215272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.215272    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4034aaa5-c9b9-4b51-b163-981403892ed6-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.652783 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.652752    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:23.668532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.668507    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"
Nov 03 08:15:23.668690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.668520    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd" event={"ID":"4034aaa5-c9b9-4b51-b163-981403892ed6","Type":"ContainerDied","Data":"18b2aad5b4519c3f9fcb27c282fae5b0230ae07fd9d2878f311f7b697aab7754"}
Nov 03 08:15:23.668690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.668578    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b2aad5b4519c3f9fcb27c282fae5b0230ae07fd9d2878f311f7b697aab7754"
Nov 03 08:15:23.697337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.697311    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"]
Nov 03 08:15:23.697973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.697960    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.698016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.697973    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.698016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.697988    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy"
Nov 03 08:15:23.698016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.697993    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy"
Nov 03 08:15:23.698016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698001    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.698016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698005    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.698016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698015    6430 cpu_manager.go:414] "containerMap: removing container" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698019    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698033    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4034aaa5-c9b9-4b51-b163-981403892ed6" containerName="revoke-keys"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698037    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4034aaa5-c9b9-4b51-b163-981403892ed6" containerName="revoke-keys"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698051    6430 cpu_manager.go:414] "containerMap: removing container" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698056    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698078    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698082    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698089    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698094    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698104    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698108    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698117    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698121    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698128    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698134    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698140    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698145    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698154    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698159    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698166    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698170    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698182    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698186    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698195    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.698244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698199    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698320    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698328    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698334    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698340    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698345    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698351    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698358    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698367    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" containerName="istio-proxy"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698375    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" containerName="istio-proxy"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698382    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4034aaa5-c9b9-4b51-b163-981403892ed6" containerName="revoke-keys"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698394    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" containerName="istio-proxy"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698403    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698409    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698414    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.699032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.698419    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.702037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.702021    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:23.707964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.707942    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-99410b\""
Nov 03 08:15:23.708212 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.708144    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-99410b-dockercfg-55spb\""
Nov 03 08:15:23.712483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.711215    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"]
Nov 03 08:15:23.740850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.740826    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"]
Nov 03 08:15:23.741633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741612    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.741633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741629    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741656    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741662    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741681    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741686    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741789    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7fad21-fe12-4d47-911d-c036b42ea3ad" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741803    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f06d813-d72f-4516-90b1-55419f11af20" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741811    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9e252c-56a1-483a-af31-b5a7eb2d1ff5" containerName="cleanup"
Nov 03 08:15:23.741814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.741819    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5714d107-1408-451e-be96-16017082f797" containerName="cleanup"
Nov 03 08:15:23.745576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.745556    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:23.754911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.754886    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"]
Nov 03 08:15:23.778633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.778610    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:23.864101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.831155    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjq8\" (UniqueName: \"kubernetes.io/projected/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-kube-api-access-9cjq8\") pod \"payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:23.864407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.864381    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c075e012-470f-4bfd-bc4c-16dd635cb65f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") " pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:23.873096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.864726    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/c075e012-470f-4bfd-bc4c-16dd635cb65f-kube-api-access-wtw7q\") pod \"payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") " pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:23.893090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.887240    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:23.998089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.996294    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:23.998089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.996560    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjq8\" (UniqueName: \"kubernetes.io/projected/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-kube-api-access-9cjq8\") pod \"payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:23.998089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.996639    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c075e012-470f-4bfd-bc4c-16dd635cb65f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") " pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:23.998089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:23.996786    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/c075e012-470f-4bfd-bc4c-16dd635cb65f-kube-api-access-wtw7q\") pod \"payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") " pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:24.009164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.005676    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c075e012-470f-4bfd-bc4c-16dd635cb65f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") " pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:24.009849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.009822    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:24.046598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.046565    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjq8\" (UniqueName: \"kubernetes.io/projected/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-kube-api-access-9cjq8\") pod \"payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:24.087666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.087635    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/c075e012-470f-4bfd-bc4c-16dd635cb65f-kube-api-access-wtw7q\") pod \"payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") " pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:24.301880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.301801    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:24.329097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.325297    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:24.381445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.362356    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:24.619831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.619761    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e73f76-be64-4daf-89ec-09cbea76457c" path="/var/lib/kubelet/pods/13e73f76-be64-4daf-89ec-09cbea76457c/volumes"
Nov 03 08:15:24.620237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.620223    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d812ab2-d05b-4bc6-b8d1-2cf70390c34e" path="/var/lib/kubelet/pods/3d812ab2-d05b-4bc6-b8d1-2cf70390c34e/volumes"
Nov 03 08:15:24.620678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.620665    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bea25b-da8a-4336-aeaf-f82b3e59d563" path="/var/lib/kubelet/pods/46bea25b-da8a-4336-aeaf-f82b3e59d563/volumes"
Nov 03 08:15:24.621480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.621464    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5231c00e-b16d-40a8-9dbe-76f1509d758b" path="/var/lib/kubelet/pods/5231c00e-b16d-40a8-9dbe-76f1509d758b/volumes"
Nov 03 08:15:24.621842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.621829    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f06d813-d72f-4516-90b1-55419f11af20" path="/var/lib/kubelet/pods/7f06d813-d72f-4516-90b1-55419f11af20/volumes"
Nov 03 08:15:24.702899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.702874    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"]
Nov 03 08:15:24.703667 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:24.703640    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc075e012_470f_4bfd_bc4c_16dd635cb65f.slice/crio-dc3e1de091b4053250a3532d1e2a538a4a1c9e69c0142d74861fd29e66f721ee WatchSource:0}: Error finding container dc3e1de091b4053250a3532d1e2a538a4a1c9e69c0142d74861fd29e66f721ee: Status 404 returned error can't find the container with id dc3e1de091b4053250a3532d1e2a538a4a1c9e69c0142d74861fd29e66f721ee
Nov 03 08:15:24.707388 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:24.707366    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e54c51d_3bfe_44dc_a4b5_7b1ac2315b5e.slice/crio-324922b351661db4f9554ca3b9fd3d3910adc60bf98355950d0ef677baf77cad WatchSource:0}: Error finding container 324922b351661db4f9554ca3b9fd3d3910adc60bf98355950d0ef677baf77cad: Status 404 returned error can't find the container with id 324922b351661db4f9554ca3b9fd3d3910adc60bf98355950d0ef677baf77cad
Nov 03 08:15:24.707505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:24.707419    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"]
Nov 03 08:15:25.563341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.563308    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:25.691837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.691797    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr" event={"ID":"c075e012-470f-4bfd-bc4c-16dd635cb65f","Type":"ContainerStarted","Data":"5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4"}
Nov 03 08:15:25.692018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.691855    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr" event={"ID":"c075e012-470f-4bfd-bc4c-16dd635cb65f","Type":"ContainerStarted","Data":"dc3e1de091b4053250a3532d1e2a538a4a1c9e69c0142d74861fd29e66f721ee"}
Nov 03 08:15:25.692018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.691884    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:25.693686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.693662    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47" event={"ID":"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e","Type":"ContainerStarted","Data":"6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24"}
Nov 03 08:15:25.693788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.693705    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47" event={"ID":"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e","Type":"ContainerStarted","Data":"324922b351661db4f9554ca3b9fd3d3910adc60bf98355950d0ef677baf77cad"}
Nov 03 08:15:25.693788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.693747    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:25.716873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.716823    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr" podStartSLOduration=2.716810662 podStartE2EDuration="2.716810662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:25.715315002 +0000 UTC m=+1275.218150516" watchObservedRunningTime="2026-11-03 08:15:25.716810662 +0000 UTC m=+1275.219646207"
Nov 03 08:15:25.743712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:25.743671    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47" podStartSLOduration=2.743657582 podStartE2EDuration="2.743657582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:25.741174288 +0000 UTC m=+1275.244009814" watchObservedRunningTime="2026-11-03 08:15:25.743657582 +0000 UTC m=+1275.246493096"
Nov 03 08:15:26.552553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:26.552516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:27.528032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:27.527544    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:28.226427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.220944    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"]
Nov 03 08:15:28.230996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.228114    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.230996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.230502    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"]
Nov 03 08:15:28.243785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.243764    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-179c16-serving-cert\""
Nov 03 08:15:28.272345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.272313    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:28.399106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.383081    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:28.399106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.397614    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dda9a6-e738-403b-8b3a-2aa22356b419-openshift-service-ca\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.399106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.397754    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-metrics-certs\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.399106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.397843    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnkn\" (UniqueName: \"kubernetes.io/projected/b7dda9a6-e738-403b-8b3a-2aa22356b419-kube-api-access-2qnkn\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.399106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.397900    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-maas-api-tls\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.460831 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:28.460718    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:15:28.499674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.499593    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dda9a6-e738-403b-8b3a-2aa22356b419-openshift-service-ca\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.499850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.499687    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-metrics-certs\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.499850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.499757    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnkn\" (UniqueName: \"kubernetes.io/projected/b7dda9a6-e738-403b-8b3a-2aa22356b419-kube-api-access-2qnkn\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.499850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.499802    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-maas-api-tls\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.500455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.500430    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dda9a6-e738-403b-8b3a-2aa22356b419-openshift-service-ca\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.503176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.503154    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-maas-api-tls\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.503338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.503164    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-metrics-certs\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.507542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.507521    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnkn\" (UniqueName: \"kubernetes.io/projected/b7dda9a6-e738-403b-8b3a-2aa22356b419-kube-api-access-2qnkn\") pod \"maas-api-e2e-inf-b-179c16-56db974668-kmkjl\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:28.510207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.510181    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:28.546545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.546515    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:29.001584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:28.993117    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:29.082818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:29.080993    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"]
Nov 03 08:15:29.180360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:29.180328    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:29.449086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:29.443777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:29.580086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:29.572596    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:29.766089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:29.753642    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl" event={"ID":"b7dda9a6-e738-403b-8b3a-2aa22356b419","Type":"ContainerStarted","Data":"4bdc795b8722709f2786957dcc51aca1b066d399c78d390ca0ceeac8bedd6cb4"}
Nov 03 08:15:30.280806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:30.280766    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:30.763012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:30.762974    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl" event={"ID":"b7dda9a6-e738-403b-8b3a-2aa22356b419","Type":"ContainerStarted","Data":"5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f"}
Nov 03 08:15:30.763526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:30.763104    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:30.781736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:30.781681    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl" podStartSLOduration=2.263756046 podStartE2EDuration="2.781663863s" totalImagesPullingTime="517.907817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:30.779270525 +0000 UTC m=+1280.282106041" watchObservedRunningTime="2026-11-03 08:15:30.781663863 +0000 UTC m=+1280.284499379"
Nov 03 08:15:31.076793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:31.076708    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:32.111700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:32.111661    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:33.099138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:33.099107    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:33.563874 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:33.563839    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:33.689879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:33.689842    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:33.887239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:33.887160    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:34.154861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.154787    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:34.311357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.311319    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:34.755722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.755688    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:34.876599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.876563    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"]
Nov 03 08:15:34.882607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.882584    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:34.888034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.888006    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"]
Nov 03 08:15:34.890243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.890221    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-27c674\""
Nov 03 08:15:34.890366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.890358    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-27c674-dockercfg-8xmh7\""
Nov 03 08:15:34.893373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.893342    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:34.926476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.926448    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"]
Nov 03 08:15:34.930921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.930899    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:34.943220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:34.943196    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"]
Nov 03 08:15:35.014133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.003424    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:35.014133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.003653    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsjs\" (UniqueName: \"kubernetes.io/projected/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-kube-api-access-9rsjs\") pod \"payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:35.166211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.109033    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsjs\" (UniqueName: \"kubernetes.io/projected/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-kube-api-access-9rsjs\") pod \"payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:35.166211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.109170    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9xw\" (UniqueName: \"kubernetes.io/projected/395b8892-0a9e-4e4a-9885-44f6c18d5c89-kube-api-access-ht9xw\") pod \"payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:35.166211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.109206    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395b8892-0a9e-4e4a-9885-44f6c18d5c89-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:35.166211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.109429    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:35.166211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.110482    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:35.304436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.212380    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9xw\" (UniqueName: \"kubernetes.io/projected/395b8892-0a9e-4e4a-9885-44f6c18d5c89-kube-api-access-ht9xw\") pod \"payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:35.304830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.304803    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395b8892-0a9e-4e4a-9885-44f6c18d5c89-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:35.320729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.305736    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395b8892-0a9e-4e4a-9885-44f6c18d5c89-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:35.369669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.366804    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9xw\" (UniqueName: \"kubernetes.io/projected/395b8892-0a9e-4e4a-9885-44f6c18d5c89-kube-api-access-ht9xw\") pod \"payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:35.377279 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.376975    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsjs\" (UniqueName: \"kubernetes.io/projected/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-kube-api-access-9rsjs\") pod \"payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:35.510161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.510133    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:35.559071 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.558964    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:35.561447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.561403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:35.805115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.803841    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"]
Nov 03 08:15:35.813786 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:35.813737    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe1ae87_a384_429c_bb08_cc0f2a37fcb8.slice/crio-243a06af110c331bc4b00532f74bac25af1a2f91502367dad5cfa51c76859f3f WatchSource:0}: Error finding container 243a06af110c331bc4b00532f74bac25af1a2f91502367dad5cfa51c76859f3f: Status 404 returned error can't find the container with id 243a06af110c331bc4b00532f74bac25af1a2f91502367dad5cfa51c76859f3f
Nov 03 08:15:35.860698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.860652    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w" event={"ID":"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8","Type":"ContainerStarted","Data":"243a06af110c331bc4b00532f74bac25af1a2f91502367dad5cfa51c76859f3f"}
Nov 03 08:15:35.999441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:35.999417    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"]
Nov 03 08:15:36.001527 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:36.001502    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395b8892_0a9e_4e4a_9885_44f6c18d5c89.slice/crio-f2a613e77dd25d291c9c507f6fed56041f705c70b2ed4bf350fb9e448d49db69 WatchSource:0}: Error finding container f2a613e77dd25d291c9c507f6fed56041f705c70b2ed4bf350fb9e448d49db69: Status 404 returned error can't find the container with id f2a613e77dd25d291c9c507f6fed56041f705c70b2ed4bf350fb9e448d49db69
Nov 03 08:15:36.158539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.158503    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:36.343142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.338374    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:36.588108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.587206    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:36.718096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.712655    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:15:36.718096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.714127    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:15:36.756842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.756797    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:36.833420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.833364    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:15:36.879101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.874420    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf" event={"ID":"395b8892-0a9e-4e4a-9885-44f6c18d5c89","Type":"ContainerStarted","Data":"86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2"}
Nov 03 08:15:36.879101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.874487    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:36.879101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.874507    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf" event={"ID":"395b8892-0a9e-4e4a-9885-44f6c18d5c89","Type":"ContainerStarted","Data":"f2a613e77dd25d291c9c507f6fed56041f705c70b2ed4bf350fb9e448d49db69"}
Nov 03 08:15:36.879101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.877280    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w" event={"ID":"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8","Type":"ContainerStarted","Data":"35771ccde7d9220d4ccdeb0df04c3717744eb7e52a39f317ebdf1e93376b7cfb"}
Nov 03 08:15:36.879101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.877426    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:36.891199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.890988    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf" podStartSLOduration=2.890975588 podStartE2EDuration="2.890975588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:36.889481104 +0000 UTC m=+1286.392316625" watchObservedRunningTime="2026-11-03 08:15:36.890975588 +0000 UTC m=+1286.393811116"
Nov 03 08:15:36.906325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:36.906272    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w" podStartSLOduration=2.90625544 podStartE2EDuration="2.90625544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:36.905385535 +0000 UTC m=+1286.408221049" watchObservedRunningTime="2026-11-03 08:15:36.90625544 +0000 UTC m=+1286.409090957"
Nov 03 08:15:37.106951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:37.106855    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:37.823618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:37.823581    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:38.671956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:38.671911    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:39.216777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:39.216743    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:39.330248 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:39.330210    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:39.577339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:39.577260    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:40.421870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:40.421836    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:40.542506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:40.542476    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:41.284198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:41.284159    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:42.145086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:42.141152    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:42.849218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:42.849181    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:43.751702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:43.751670    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:43.878579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:43.878548    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:43.996084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:43.995758    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"]
Nov 03 08:15:44.034227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.031766    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"]
Nov 03 08:15:44.034227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.031822    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"]
Nov 03 08:15:44.034227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.032026    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy" containerID="cri-o://8e78bc2774c8c1e712558d59b2447306638ca7bef149f7a6439da15c5c27d99e" gracePeriod=30
Nov 03 08:15:44.034227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.032507    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.066145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.066113    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-99410b-serving-cert\""
Nov 03 08:15:44.104976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.104928    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:44.233084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.232583    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh47t\" (UniqueName: \"kubernetes.io/projected/b66d4820-51da-4287-9ef0-3d27b30c63d3-kube-api-access-wh47t\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.233084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.232652    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-maas-api-tls\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.233084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.232932    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-metrics-certs\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.233084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.232976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66d4820-51da-4287-9ef0-3d27b30c63d3-openshift-service-ca\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.256088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.254989    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:44.345135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.339595    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-metrics-certs\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.345135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.339663    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66d4820-51da-4287-9ef0-3d27b30c63d3-openshift-service-ca\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.345135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.339763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh47t\" (UniqueName: \"kubernetes.io/projected/b66d4820-51da-4287-9ef0-3d27b30c63d3-kube-api-access-wh47t\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.345135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.339799    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-maas-api-tls\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.345135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.341566    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66d4820-51da-4287-9ef0-3d27b30c63d3-openshift-service-ca\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.358015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.357977    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-metrics-certs\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.373245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.358708    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-maas-api-tls\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.392118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.386282    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:44.393905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.393873    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh47t\" (UniqueName: \"kubernetes.io/projected/b66d4820-51da-4287-9ef0-3d27b30c63d3-kube-api-access-wh47t\") pod \"maas-api-e2e-api-b-99410b-b86c85857-q8p8r\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:44.729107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:44.726414    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:45.311528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:45.311494    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"]
Nov 03 08:15:45.312230 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:15:45.312202    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66d4820_51da_4287_9ef0_3d27b30c63d3.slice/crio-be691e199b00f1cdd546e712ba6dc7cc8764a48560eabcb01c0ba8835ec6b217 WatchSource:0}: Error finding container be691e199b00f1cdd546e712ba6dc7cc8764a48560eabcb01c0ba8835ec6b217: Status 404 returned error can't find the container with id be691e199b00f1cdd546e712ba6dc7cc8764a48560eabcb01c0ba8835ec6b217
Nov 03 08:15:45.432296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:45.432266    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:45.587592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:45.587503    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:45.810585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:45.810545    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:46.099574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:46.099523    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r" event={"ID":"b66d4820-51da-4287-9ef0-3d27b30c63d3","Type":"ContainerStarted","Data":"afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81"}
Nov 03 08:15:46.099752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:46.099597    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r" event={"ID":"b66d4820-51da-4287-9ef0-3d27b30c63d3","Type":"ContainerStarted","Data":"be691e199b00f1cdd546e712ba6dc7cc8764a48560eabcb01c0ba8835ec6b217"}
Nov 03 08:15:46.099752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:46.099628    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:46.120917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:46.120868    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r" podStartSLOduration=2.728657161 podStartE2EDuration="3.120854165s" totalImagesPullingTime="392.197004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:46.117266001 +0000 UTC m=+1295.620101526" watchObservedRunningTime="2026-11-03 08:15:46.120854165 +0000 UTC m=+1295.623689678"
Nov 03 08:15:46.248827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:46.248727    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:46.402625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:46.402541    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:46.540093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:46.540002    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:47.559900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:47.559828    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:47.927248 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:47.927173    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:15:47.927665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:47.927645    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:15:48.437296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:48.437236    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:49.145630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.145600    6430 generic.go:356] "Generic (PLEG): container finished" podID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerID="8e78bc2774c8c1e712558d59b2447306638ca7bef149f7a6439da15c5c27d99e" exitCode=0
Nov 03 08:15:49.146111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.145673    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" event={"ID":"d62d7575-8abc-452e-ba8d-b0965e6f683e","Type":"ContainerDied","Data":"8e78bc2774c8c1e712558d59b2447306638ca7bef149f7a6439da15c5c27d99e"}
Nov 03 08:15:49.146111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.145736    6430 scope.go:122] "RemoveContainer" containerID="d29585134e5b7b1446dfffd90579c9a48444dad017e89d6f44506c503a69db70"
Nov 03 08:15:49.254205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.254182    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:15:49.448492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448455    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-podinfo\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.448655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448505    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-credential-socket\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.448655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448581    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-socket\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.448779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448681    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-token\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-token\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.448779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448710    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-certs\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.448888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448839    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-credential-socket" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:49.448888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448856    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62d7575-8abc-452e-ba8d-b0965e6f683e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d7575-8abc-452e-ba8d-b0965e6f683e-istiod-ca-cert\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.449000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448886    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-socket" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:49.449000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448913    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-kube-api-access-gb5pf\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-kube-api-access-gb5pf\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.449000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.448970    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-data\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.449223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449037    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-certs" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:49.449223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449046    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-envoy\") pod \"d62d7575-8abc-452e-ba8d-b0965e6f683e\" (UID: \"d62d7575-8abc-452e-ba8d-b0965e6f683e\") "
Nov 03 08:15:49.449329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449307    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-data" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:49.449383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449332    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62d7575-8abc-452e-ba8d-b0965e6f683e-istiod-ca-cert" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:49.449684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449661    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d7575-8abc-452e-ba8d-b0965e6f683e-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.449832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449686    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.449832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449696    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.449832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449705    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.449832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.449713    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.450906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.450877    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-podinfo" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:15:49.451377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.451357    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-envoy" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:49.451602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.451584    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-token" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:49.451663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.451599    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-kube-api-access-gb5pf" pod "d62d7575-8abc-452e-ba8d-b0965e6f683e" (UID: "d62d7575-8abc-452e-ba8d-b0965e6f683e"). InnerVolumeSpecName "kube-api-access-gb5pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:49.458963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.458938    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:49.554611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.554573    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.554841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.554825    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5pf\" (UniqueName: \"kubernetes.io/projected/d62d7575-8abc-452e-ba8d-b0965e6f683e-kube-api-access-gb5pf\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.554936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.554925    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:49.555020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:49.555010    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d7575-8abc-452e-ba8d-b0965e6f683e-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:50.045485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.045456    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:50.155568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.155516    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"
Nov 03 08:15:50.155568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.155531    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz" event={"ID":"d62d7575-8abc-452e-ba8d-b0965e6f683e","Type":"ContainerDied","Data":"e01a418d49fcc6d0785ef7ee817ce8e4839cad2b0ae2f9c627e6dca12a8e1e21"}
Nov 03 08:15:50.156048 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.155592    6430 scope.go:122] "RemoveContainer" containerID="8e78bc2774c8c1e712558d59b2447306638ca7bef149f7a6439da15c5c27d99e"
Nov 03 08:15:50.172236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.172186    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:50.182391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.182366    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"]
Nov 03 08:15:50.185775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.185754    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-94e3cb-openshift-default-6cd8645f69-jp8qz"]
Nov 03 08:15:50.555544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.555507    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:50.621817 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:50.621784    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" path="/var/lib/kubelet/pods/d62d7575-8abc-452e-ba8d-b0965e6f683e/volumes"
Nov 03 08:15:51.295016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.294986    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:51.313104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.313045    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"]
Nov 03 08:15:51.313798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.313784    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy"
Nov 03 08:15:51.313844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.313800    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy"
Nov 03 08:15:51.313844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.313812    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy"
Nov 03 08:15:51.313844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.313818    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy"
Nov 03 08:15:51.313951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.313941    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy"
Nov 03 08:15:51.313982 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.313952    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62d7575-8abc-452e-ba8d-b0965e6f683e" containerName="istio-proxy"
Nov 03 08:15:51.319154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.319136    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:51.327980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.327955    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-616e46\""
Nov 03 08:15:51.330875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.330853    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"]
Nov 03 08:15:51.383350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.383320    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:51.383567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.383441    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7bcz\" (UniqueName: \"kubernetes.io/projected/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-kube-api-access-g7bcz\") pod \"payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:51.417956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.417866    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:51.490161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.490118    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7bcz\" (UniqueName: \"kubernetes.io/projected/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-kube-api-access-g7bcz\") pod \"payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:51.497119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.490475    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:51.497119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.491445    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:51.553400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.553119    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7bcz\" (UniqueName: \"kubernetes.io/projected/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-kube-api-access-g7bcz\") pod \"payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:51.639085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:51.638399    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:52.173133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:52.168220    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:15:52.175508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:52.175485    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"]
Nov 03 08:15:52.591280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:52.591242    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j" event={"ID":"d0fd224e-e7a1-46be-89cf-3fae1f1120e2","Type":"ContainerStarted","Data":"b75d609280f96c177cde807a7939893410a3228b7da80dfad9dbf9c30304a394"}
Nov 03 08:15:52.853413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:52.853310    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:53.699083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:53.697767    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j" event={"ID":"d0fd224e-e7a1-46be-89cf-3fae1f1120e2","Type":"ContainerStarted","Data":"f72271a77abd74d9927de01f733c9d303b94377e1687d0db7047d3990118ef08"}
Nov 03 08:15:53.699083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:53.699043    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:15:53.730912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:53.730581    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j" podStartSLOduration=2.730560466 podStartE2EDuration="2.730560466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:53.721641271 +0000 UTC m=+1303.224476788" watchObservedRunningTime="2026-11-03 08:15:53.730560466 +0000 UTC m=+1303.233395982"
Nov 03 08:15:54.247514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:54.247475    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:54.947720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:54.947686    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:55.070539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:55.070503    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:56.133198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:56.133169    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:56.551044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:56.551009    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:57.320128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:57.320095    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:58.235398 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:58.235327    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:58.350943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:58.350871    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:58.485876 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:15:58.484706    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:15:59.447365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.447332    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"]
Nov 03 08:15:59.452419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.452398    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.473304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.473279    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ba69a0\"/\"kube-root-ca.crt\""
Nov 03 08:15:59.474642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.474005    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ba69a0\"/\"openshift-service-ca.crt\""
Nov 03 08:15:59.474642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.474325    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ba69a0\"/\"default-dockercfg-qkd8d\""
Nov 03 08:15:59.474642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.474525    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ba69a0\"/\"test-model-ba69a0-kserve-self-signed-certs\""
Nov 03 08:15:59.477045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.477021    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"]
Nov 03 08:15:59.553893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.553858    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95sg\" (UniqueName: \"kubernetes.io/projected/c62a3394-de42-4cda-b192-8157f006be53-kube-api-access-m95sg\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.554089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.553956    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-tmp-dir\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.554089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.554045    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-model-cache\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.554222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.554204    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-home\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.554281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.554232    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-dshm\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.554281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.554273    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a3394-de42-4cda-b192-8157f006be53-tls-certs\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.582547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.582512    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:15:59.656415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.656382    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-tmp-dir\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.656622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.656499    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-model-cache\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.656622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.656616    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-home\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.656743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.656667    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-dshm\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.656951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.656870    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a3394-de42-4cda-b192-8157f006be53-tls-certs\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.656951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.656885    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-tmp-dir\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.657149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.656957    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-model-cache\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.657149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.657129    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95sg\" (UniqueName: \"kubernetes.io/projected/c62a3394-de42-4cda-b192-8157f006be53-kube-api-access-m95sg\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.657484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.657441    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-home\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.659451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.659425    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-dshm\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.659645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.659627    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a3394-de42-4cda-b192-8157f006be53-tls-certs\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.664465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.664444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95sg\" (UniqueName: \"kubernetes.io/projected/c62a3394-de42-4cda-b192-8157f006be53-kube-api-access-m95sg\") pod \"test-model-ba69a0-kserve-d5b5475cc-zw5ln\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") " pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.770380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.770292    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:15:59.811708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:15:59.811667    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:00.263015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:00.262986    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"]
Nov 03 08:16:00.264892 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:16:00.264863    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc62a3394_de42_4cda_b192_8157f006be53.slice/crio-6f09b1faf327f4b795a92e9775476cb2c275088020d18233b72126503f61bac5 WatchSource:0}: Error finding container 6f09b1faf327f4b795a92e9775476cb2c275088020d18233b72126503f61bac5: Status 404 returned error can't find the container with id 6f09b1faf327f4b795a92e9775476cb2c275088020d18233b72126503f61bac5
Nov 03 08:16:00.801195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:00.801160    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:00.905592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:00.903820    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" event={"ID":"c62a3394-de42-4cda-b192-8157f006be53","Type":"ContainerStarted","Data":"f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7"}
Nov 03 08:16:00.905592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:00.903883    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" event={"ID":"c62a3394-de42-4cda-b192-8157f006be53","Type":"ContainerStarted","Data":"6f09b1faf327f4b795a92e9775476cb2c275088020d18233b72126503f61bac5"}
Nov 03 08:16:00.923792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:00.923740    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" podStartSLOduration=1.923725418 podStartE2EDuration="1.923725418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:00.922004429 +0000 UTC m=+1310.424839934" watchObservedRunningTime="2026-11-03 08:16:00.923725418 +0000 UTC m=+1310.426561126"
Nov 03 08:16:01.667260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:01.667222    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:01.776260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:01.771428    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:16:02.655229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:02.655187    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:03.461182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:03.461149    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:04.390149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:04.390108    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:04.525096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:04.524587    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:05.393464 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:05.393435    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:05.720256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:05.720226    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:16:06.266784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:06.266747    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:07.272630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:07.272593    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:08.051495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:08.051458    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:09.025778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:09.025746    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:09.771235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:09.771200    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:16:09.784267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:09.784235    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:16:09.990425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:09.990400    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:10.100791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.100711    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:16:10.881037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.880999    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"]
Nov 03 08:16:10.911104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.910794    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"]
Nov 03 08:16:10.911104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.910910    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.949979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.949950    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b9ca2-2706-4449-8102-55b7b146379e-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950009    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b9ca2-2706-4449-8102-55b7b146379e-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950227    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950330    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950442    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950487    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950646    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950690    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.950805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.950790    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vzz\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-kube-api-access-t8vzz\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:10.956860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:10.956831    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e0f576-openshift-default-dockercfg-wnxjc\""
Nov 03 08:16:11.008813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.008780    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"]
Nov 03 08:16:11.022087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.021198    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.025581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.025555    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"]
Nov 03 08:16:11.029426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.029401    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-179c16\"/\"default-dockercfg-4m985\""
Nov 03 08:16:11.029580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.029566    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-179c16\"/\"test-model-179c16-kserve-self-signed-certs\""
Nov 03 08:16:11.029669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.029639    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-179c16\"/\"kube-root-ca.crt\""
Nov 03 08:16:11.029966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.029948    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-179c16\"/\"openshift-service-ca.crt\""
Nov 03 08:16:11.054902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.054781    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b9ca2-2706-4449-8102-55b7b146379e-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.054902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.054859    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b9ca2-2706-4449-8102-55b7b146379e-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.056012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.055930    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b9ca2-2706-4449-8102-55b7b146379e-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.056012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.055943    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:11.056821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.056795    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.056926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.056862    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.056992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.056955    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.057052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.056994    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.057172 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.057156    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.057244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.057198    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.057305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.057259    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vzz\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-kube-api-access-t8vzz\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.059084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.058811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.059323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.059299    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.059652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.059629    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.060225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.060204    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.064603 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.064572    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b9ca2-2706-4449-8102-55b7b146379e-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.064766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.064745    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.066851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.066830    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vzz\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-kube-api-access-t8vzz\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.067432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.067401    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.158370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.158277    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-dshm\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.158370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.158346    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-home\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.158889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.158412    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntrr\" (UniqueName: \"kubernetes.io/projected/10f80321-1867-4c5d-8462-24a3a17c4831-kube-api-access-pntrr\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.158889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.158521    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-model-cache\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.158889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.158622    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80321-1867-4c5d-8462-24a3a17c4831-tls-certs\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.158889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.158700    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-tmp-dir\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.179952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.179916    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:11.253839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.252440    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:11.268722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.268153    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntrr\" (UniqueName: \"kubernetes.io/projected/10f80321-1867-4c5d-8462-24a3a17c4831-kube-api-access-pntrr\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.268722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.268210    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-model-cache\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.268722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.268365    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80321-1867-4c5d-8462-24a3a17c4831-tls-certs\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.268722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.268413    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-tmp-dir\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.268722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.268548    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-dshm\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.268722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.268626    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-home\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.269170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.269129    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-home\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.286108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.274170    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-tmp-dir\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.286108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.274902    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-model-cache\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.299196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.299153    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80321-1867-4c5d-8462-24a3a17c4831-tls-certs\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.383866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.363804    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-dshm\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.435135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.433790    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntrr\" (UniqueName: \"kubernetes.io/projected/10f80321-1867-4c5d-8462-24a3a17c4831-kube-api-access-pntrr\") pod \"test-model-179c16-kserve-55c658c8df-zz4dk\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") " pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.659019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.658998    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:11.763031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.762998    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"]
Nov 03 08:16:11.764732 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:16:11.764699    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43b9ca2_2706_4449_8102_55b7b146379e.slice/crio-dc8d870362959e480aed71f00a272b99d8327df6fa8b2f0a0d5fe7ad41c1741c WatchSource:0}: Error finding container dc8d870362959e480aed71f00a272b99d8327df6fa8b2f0a0d5fe7ad41c1741c: Status 404 returned error can't find the container with id dc8d870362959e480aed71f00a272b99d8327df6fa8b2f0a0d5fe7ad41c1741c
Nov 03 08:16:11.767220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.767181    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:11.767327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.767253    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:11.767327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.767296    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:11.943679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:11.943638    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"]
Nov 03 08:16:11.947150 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:16:11.947105    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f80321_1867_4c5d_8462_24a3a17c4831.slice/crio-72c60b65c420d7a19d6b1681e5bb998b300bf3e5313bf9e51dc9e509a951f355 WatchSource:0}: Error finding container 72c60b65c420d7a19d6b1681e5bb998b300bf3e5313bf9e51dc9e509a951f355: Status 404 returned error can't find the container with id 72c60b65c420d7a19d6b1681e5bb998b300bf3e5313bf9e51dc9e509a951f355
Nov 03 08:16:12.057078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.056050    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk" event={"ID":"10f80321-1867-4c5d-8462-24a3a17c4831","Type":"ContainerStarted","Data":"16714577501c69c98d0d3d7fb6bc43dac33b043d5737ae152b0df344d252866a"}
Nov 03 08:16:12.057078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.056113    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk" event={"ID":"10f80321-1867-4c5d-8462-24a3a17c4831","Type":"ContainerStarted","Data":"72c60b65c420d7a19d6b1681e5bb998b300bf3e5313bf9e51dc9e509a951f355"}
Nov 03 08:16:12.059074 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.058762    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" event={"ID":"f43b9ca2-2706-4449-8102-55b7b146379e","Type":"ContainerStarted","Data":"8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c"}
Nov 03 08:16:12.059074 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.058852    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" event={"ID":"f43b9ca2-2706-4449-8102-55b7b146379e","Type":"ContainerStarted","Data":"dc8d870362959e480aed71f00a272b99d8327df6fa8b2f0a0d5fe7ad41c1741c"}
Nov 03 08:16:12.073790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.073737    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk" podStartSLOduration=2.073720597 podStartE2EDuration="2.073720597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:12.071462401 +0000 UTC m=+1321.574297916" watchObservedRunningTime="2026-11-03 08:16:12.073720597 +0000 UTC m=+1321.576556113"
Nov 03 08:16:12.095206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.095147    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podStartSLOduration=2.095126361 podStartE2EDuration="2.095126361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:12.089498745 +0000 UTC m=+1321.592334260" watchObservedRunningTime="2026-11-03 08:16:12.095126361 +0000 UTC m=+1321.597961876"
Nov 03 08:16:12.257842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.253427    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:12.271084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.265180    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:12.271084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.265264    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:12.659213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.659134    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:12.707387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.707358    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:12.831884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:12.831822    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:13.253386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:13.253352    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:13.253564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:13.253412    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:13.854549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:13.854513    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:14.254205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.254160    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:14.254411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.254239    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:14.334858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.334825    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"]
Nov 03 08:16:14.339734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.339712    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.351344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.351313    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9b559e-openshift-default-dockercfg-6hqmb\""
Nov 03 08:16:14.355515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.355489    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"]
Nov 03 08:16:14.391895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.391860    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4r4\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-kube-api-access-zc4r4\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.391989    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83537c3e-e723-4150-a413-c817d9630222-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.392024    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.392112    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.392242    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.392330    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.392385    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83537c3e-e723-4150-a413-c817d9630222-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.392441    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.392531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.392462    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.493693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.493657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.493860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.493756    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.493860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.493823    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83537c3e-e723-4150-a413-c817d9630222-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.493969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.493862    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.493969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.493889    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494018    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4r4\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-kube-api-access-zc4r4\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494152    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83537c3e-e723-4150-a413-c817d9630222-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494205    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494232    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494240    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494266    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494554 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494537    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.494716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.494667    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.495115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.495092    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83537c3e-e723-4150-a413-c817d9630222-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.496765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.496742    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83537c3e-e723-4150-a413-c817d9630222-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.496882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.496763    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.531120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.530771    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4r4\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-kube-api-access-zc4r4\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.534495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.534459    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.656567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.656539    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:14.980025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.979995    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"]
Nov 03 08:16:14.982417 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:16:14.982385    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83537c3e_e723_4150_a413_c817d9630222.slice/crio-2ee6991ae2d56fcb389a9f0901edff59b01de8db1cb17d15e0973dbeb208305d WatchSource:0}: Error finding container 2ee6991ae2d56fcb389a9f0901edff59b01de8db1cb17d15e0973dbeb208305d: Status 404 returned error can't find the container with id 2ee6991ae2d56fcb389a9f0901edff59b01de8db1cb17d15e0973dbeb208305d
Nov 03 08:16:14.985755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.985713    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:14.985856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.985805    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:14.985924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:14.985896    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:15.092020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.091994    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" event={"ID":"83537c3e-e723-4150-a413-c817d9630222","Type":"ContainerStarted","Data":"2ee6991ae2d56fcb389a9f0901edff59b01de8db1cb17d15e0973dbeb208305d"}
Nov 03 08:16:15.155287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.154787    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"]
Nov 03 08:16:15.178653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.176321    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.178653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.177593    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"]
Nov 03 08:16:15.187215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.182505    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-228695-openshift-default-dockercfg-9dgpq\""
Nov 03 08:16:15.242308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242203    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242307    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242360    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242390    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242445    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242479    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242516    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9r2\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-kube-api-access-5d9r2\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242645    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.242822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.242744    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.255019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.253981    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:15.255019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.254040    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:15.344170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344128    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344189    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344217    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344231    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344255    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344275    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344305    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9r2\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-kube-api-access-5d9r2\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344386    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344432    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344591    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344629    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.344715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.344702    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.345046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.345024    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.345152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.345134    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.346863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.346839    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.347113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.347095    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.352324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.352303    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.352775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.352751    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9r2\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-kube-api-access-5d9r2\") pod \"e2e-api-b-228695-openshift-default-547df686cf-rlcl4\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.556676 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.556603    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:15.933732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.933699    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"]
Nov 03 08:16:15.933917 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:16:15.933681    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c73801_07d7_42bb_9f3f_6d3c2c0b6953.slice/crio-006f1054f4584a560f7bb2231fbb1b04a2ddf9cb85e576c1a6abb83d6bfb9a9b WatchSource:0}: Error finding container 006f1054f4584a560f7bb2231fbb1b04a2ddf9cb85e576c1a6abb83d6bfb9a9b: Status 404 returned error can't find the container with id 006f1054f4584a560f7bb2231fbb1b04a2ddf9cb85e576c1a6abb83d6bfb9a9b
Nov 03 08:16:15.937482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.937454    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:15.941188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.941158    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:15.941359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:15.941342    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:16.153089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.152696    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" event={"ID":"83537c3e-e723-4150-a413-c817d9630222","Type":"ContainerStarted","Data":"09b6aa980cc2fd1a8559afa4070f259f58d167561f4475ace4f58d09b6fd03da"}
Nov 03 08:16:16.156305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.156270    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" event={"ID":"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953","Type":"ContainerStarted","Data":"b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a"}
Nov 03 08:16:16.156474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.156333    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" event={"ID":"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953","Type":"ContainerStarted","Data":"006f1054f4584a560f7bb2231fbb1b04a2ddf9cb85e576c1a6abb83d6bfb9a9b"}
Nov 03 08:16:16.178088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.176109    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podStartSLOduration=2.176094245 podStartE2EDuration="2.176094245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:16.171638022 +0000 UTC m=+1325.674473536" watchObservedRunningTime="2026-11-03 08:16:16.176094245 +0000 UTC m=+1325.678929755"
Nov 03 08:16:16.195477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.195128    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podStartSLOduration=1.195113869 podStartE2EDuration="1.195113869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:16.19095996 +0000 UTC m=+1325.693795474" watchObservedRunningTime="2026-11-03 08:16:16.195113869 +0000 UTC m=+1325.697949382"
Nov 03 08:16:16.216098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.214572    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:16.258119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.255513    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:16.258119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.255589    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:16.355133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.351574    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:16.563539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.559163    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:16.659413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.659381    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 08:16:16.659623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.659439    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 08:16:16.660563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.660544    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:16.926142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:16.924082    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"]
Nov 03 08:16:17.021711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.021622    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:17.079281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.079233    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"]
Nov 03 08:16:17.103729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.079668    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.271294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.184920    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-616e46-serving-cert\""
Nov 03 08:16:17.271294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.189980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcdw\" (UniqueName: \"kubernetes.io/projected/4bde8834-c6f6-44d8-8df3-542c155b7315-kube-api-access-zwcdw\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.271294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.190194    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-maas-api-tls\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.271294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.190339    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-metrics-certs\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.271294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.190413    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde8834-c6f6-44d8-8df3-542c155b7315-openshift-service-ca\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.271294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.265679    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:17.314197 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.273796    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:17.314197 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.273863    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:17.326118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.322598    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcdw\" (UniqueName: \"kubernetes.io/projected/4bde8834-c6f6-44d8-8df3-542c155b7315-kube-api-access-zwcdw\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.326118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.322812    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-maas-api-tls\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.326118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.322976    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-metrics-certs\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.326118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.323053    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde8834-c6f6-44d8-8df3-542c155b7315-openshift-service-ca\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.326118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.324083    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde8834-c6f6-44d8-8df3-542c155b7315-openshift-service-ca\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.530134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.520322    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcdw\" (UniqueName: \"kubernetes.io/projected/4bde8834-c6f6-44d8-8df3-542c155b7315-kube-api-access-zwcdw\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.530134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.521310    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-metrics-certs\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.558423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.558386    6430 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-rlcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 08:16:17.559235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.559198    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 08:16:17.574719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.559138    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-maas-api-tls\") pod \"maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.691091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.668087    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 08:16:17.691091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.668157    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 08:16:17.781135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.765278    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:17.970495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:17.970453    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:18.022255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.022215    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" podUID="c62a3394-de42-4cda-b192-8157f006be53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": context deadline exceeded"
Nov 03 08:16:18.035228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.035132    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:18.074521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.074427    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:18.257089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.256827    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:18.257089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.256892    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:18.395363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.395252    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:18.551126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.513433    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:18.551126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.513791    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 08:16:18.551126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.514088    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 08:16:18.551126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.514326    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 08:16:18.651879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.621137    6430 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-rlcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 08:16:18.651879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.621197    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 08:16:18.658875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.658456    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 08:16:18.658875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:18.658509    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 08:16:19.026242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.026191    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:19.121087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.074297    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:16:19.121087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.088169    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:19.189175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.189149    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"]
Nov 03 08:16:19.189419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.189406    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:16:19.194213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.194177    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:16:19.262122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.261452    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 08:16:19.262122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.261513    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 08:16:19.567675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.567634    6430 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-rlcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 08:16:19.572228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.572169    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 08:16:19.665088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.661857    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 08:16:19.665088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:19.661920    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 08:16:20.174745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.174652    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls" event={"ID":"4bde8834-c6f6-44d8-8df3-542c155b7315","Type":"ContainerStarted","Data":"8d020769d116d1030011c1c2321b3b9dcbbb7ce28d1b92c1feb2678066d8c03b"}
Nov 03 08:16:20.174745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.174715    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls" event={"ID":"4bde8834-c6f6-44d8-8df3-542c155b7315","Type":"ContainerStarted","Data":"19d058a805805d96b465248802c44e12c2f402a6fea13400d71face8f4c9ec31"}
Nov 03 08:16:20.175898 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.175707    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:20.200092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.195786    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls" podStartSLOduration=3.6093263220000003 podStartE2EDuration="4.195772296s" totalImagesPullingTime="586.445974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:20.19028423 +0000 UTC m=+1329.693119744" watchObservedRunningTime="2026-11-03 08:16:20.195772296 +0000 UTC m=+1329.698607809"
Nov 03 08:16:20.261146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.261116    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:20.261570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.261549    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:20.262238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.262222    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:16:20.460106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.459531    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:20.560380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.557458    6430 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-rlcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 08:16:20.560380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.559796    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 08:16:20.729157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.670328    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 08:16:20.729157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:20.670389    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 08:16:21.557532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:21.557494    6430 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-rlcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 08:16:21.557735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:21.557582    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 08:16:21.657538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:21.657501    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 08:16:21.657834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:21.657797    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 08:16:21.659724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:21.659696    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:21.673308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:21.673286    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:21.761799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:21.761712    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:22.258438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:22.258405    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:16:22.557282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:22.557204    6430 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-rlcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 08:16:22.557282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:22.557255    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 08:16:22.696074 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:22.696022    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:22.696839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:22.696817    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:22.733095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:22.732448    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:16:22.786666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:22.786625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:23.563302 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:23.563272    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:23.563786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:23.563764    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:23.564606 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:23.564586    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:16:23.646808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:23.646720    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:24.442881 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:24.442847    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:25.267093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:25.267046    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:25.955185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:25.953156    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"]
Nov 03 08:16:25.979119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:25.977028    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.134095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.129590    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"]
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.153415    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.153514    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.153631    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4bz\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-kube-api-access-km4bz\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.153724    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.153756    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.153863    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a857c-1664-4657-ab11-9b5ee33baa24-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.153976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.154082    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.164092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.154128    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.205176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.193824    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:16:26.242645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.240103    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:26.260767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.260558    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.260767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.260656    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4bz\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-kube-api-access-km4bz\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.260767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.260722    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.260767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.260767    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.261213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.260865    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a857c-1664-4657-ab11-9b5ee33baa24-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.261213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.260963    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.261213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.261053    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.261213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.261113    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.261436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.261217    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.261865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.261704    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.262767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.262743    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a857c-1664-4657-ab11-9b5ee33baa24-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.264123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.264082    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.264556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.264537    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.264724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.264539    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.266789 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.266296    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.271463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.267140    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.271463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.271327    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.271580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.271473    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4bz\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-kube-api-access-km4bz\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.488328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.488289    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:26.783004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.782295    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"]
Nov 03 08:16:26.785648 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:16:26.785619    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35a857c_1664_4657_ab11_9b5ee33baa24.slice/crio-b10ce251b6b9c3861fd7b9af7700158c34c59860847807fdeec686e0e9a177de WatchSource:0}: Error finding container b10ce251b6b9c3861fd7b9af7700158c34c59860847807fdeec686e0e9a177de: Status 404 returned error can't find the container with id b10ce251b6b9c3861fd7b9af7700158c34c59860847807fdeec686e0e9a177de
Nov 03 08:16:26.806524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.805527    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:26.806524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.805692    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:26.806524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:26.805789    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:27.248875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:27.248835    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" event={"ID":"e35a857c-1664-4657-ab11-9b5ee33baa24","Type":"ContainerStarted","Data":"321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8"}
Nov 03 08:16:27.249321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:27.248926    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" event={"ID":"e35a857c-1664-4657-ab11-9b5ee33baa24","Type":"ContainerStarted","Data":"b10ce251b6b9c3861fd7b9af7700158c34c59860847807fdeec686e0e9a177de"}
Nov 03 08:16:27.324102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:27.322471    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podStartSLOduration=2.32244659 podStartE2EDuration="2.32244659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:27.294670725 +0000 UTC m=+1336.797506243" watchObservedRunningTime="2026-11-03 08:16:27.32244659 +0000 UTC m=+1336.825282105"
Nov 03 08:16:27.327971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:27.327938    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:27.488666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:27.488630    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:27.490667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:27.490642    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 08:16:27.490788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:27.490689    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 08:16:28.167843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:28.167809    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:28.384439 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:16:28.384405    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:16:28.489157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:28.489123    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 08:16:28.489365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:28.489182    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 08:16:29.022792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.022682    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:29.367993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.367914    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"]
Nov 03 08:16:29.373705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.373670    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.390113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.389881    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"]
Nov 03 08:16:29.441385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441344    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441459    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441572    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441624    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441651    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441681    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441700    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7zm\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-kube-api-access-rj7zm\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441727    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.441772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.441755    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.488731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.488700    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 08:16:29.488890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.488750    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 08:16:29.493031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.492997    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.573615    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.576650    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.584331    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.584764    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.584860    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.584914    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7zm\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-kube-api-access-rj7zm\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.584975    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.585024    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.585280    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.585423    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.585633    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.586715    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.588541    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.589926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.589170    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.621139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.617335    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.624717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.624686    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7zm\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-kube-api-access-rj7zm\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.653272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.652284    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.653272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.653082    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.734043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.734010    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:29.924541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:29.924110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:30.080095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.062479    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:30.129111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.129055    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"]
Nov 03 08:16:30.138044 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:16:30.137976    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb640fd_a708_4f4f_8607_ec9e414ad41a.slice/crio-bacd809281635656d5dd2b6e83c8cacf61bbff9c3fc07181fa65edb579dd7a40 WatchSource:0}: Error finding container bacd809281635656d5dd2b6e83c8cacf61bbff9c3fc07181fa65edb579dd7a40: Status 404 returned error can't find the container with id bacd809281635656d5dd2b6e83c8cacf61bbff9c3fc07181fa65edb579dd7a40
Nov 03 08:16:30.141813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.141778    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:30.141895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.141860    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:30.141942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.141899    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:16:30.345822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.345786    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" event={"ID":"5eb640fd-a708-4f4f-8607-ec9e414ad41a","Type":"ContainerStarted","Data":"cb15341d1b72c99408df5ea6b7a6df10d882e5f309330d0655f9cbeb14a95b33"}
Nov 03 08:16:30.347204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.347176    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" event={"ID":"5eb640fd-a708-4f4f-8607-ec9e414ad41a","Type":"ContainerStarted","Data":"bacd809281635656d5dd2b6e83c8cacf61bbff9c3fc07181fa65edb579dd7a40"}
Nov 03 08:16:30.357227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.354384    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:30.513137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.503419    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 08:16:30.513137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.503495    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 08:16:30.793613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:30.793537    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:31.489135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:31.489102    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 08:16:31.489312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:31.489151    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 08:16:31.735963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:31.735932    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:31.736499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:31.735981    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:32.489539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:32.489506    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 08:16:32.489696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:32.489558    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 08:16:32.734618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:32.734583    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:32.734790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:32.734635    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:33.495599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:33.495566    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:33.496144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:33.496106    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:33.497016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:33.496998    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:16:33.514788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:33.514744    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podStartSLOduration=4.514731608 podStartE2EDuration="4.514731608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:30.378161782 +0000 UTC m=+1339.880997297" watchObservedRunningTime="2026-11-03 08:16:33.514731608 +0000 UTC m=+1343.017567121"
Nov 03 08:16:33.709461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:33.709430    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:33.734731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:33.734695    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:33.734916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:33.734751    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:34.099868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:34.099804    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:34.734702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:34.734667    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:34.735184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:34.734725    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:35.745988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:35.738498    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:35.745988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:35.738580    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:36.530640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:36.530604    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:36.661085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:36.657888    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:36.737124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:36.736508    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:36.737124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:36.736588    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:37.734490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:37.734453    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:37.734910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:37.734533    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:38.735315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:38.735280    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:38.735813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:38.735337    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:39.734566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:39.734531    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:39.734748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:39.734594    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:40.734653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:40.734619    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:16:40.735023 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:40.734669    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 08:16:41.740588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:41.740561    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:41.741023 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:41.740941    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:41.741820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:41.741802    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:16:42.282340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:42.282312    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:51.551261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:51.543207    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:57.939793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:57.939745    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:58.377773 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:16:58.377688    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:16:58.797905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:58.797869    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:16:59.509085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:16:59.507986    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:00.423663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:00.423587    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:01.259519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:01.259487    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:02.081911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:02.081877    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:02.779897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:02.779868    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:03.544975 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:03.544908    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:04.245592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:04.245539    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:05.075103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:05.075053    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:05.929784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:05.929754    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:06.464956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:06.464921    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:06.579861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:06.579807    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:06.715739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:06.715655    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:07.124997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:07.124882    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:07.348332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:07.346199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:07.472228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:07.472194    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:07.686750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:07.686715    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:07.819129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:07.818947    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:07.946601 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:07.946564    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:08.094749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:08.094658    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:08.843097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:08.835028    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:09.540630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:09.540599    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:10.380247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:10.380219    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:11.229633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:11.229593    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:11.790178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:11.790144    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:12.220107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:12.220050    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:13.116614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:13.116582    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:13.452349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:13.452319    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:14.021511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:14.021468    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:15.080318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.080267    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:15.237622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.237586    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:15.354873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.354794    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"]
Nov 03 08:17:15.355340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.355307    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy" containerID="cri-o://b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a" gracePeriod=30
Nov 03 08:17:15.389455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.389038    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:15.886080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.709999    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"]
Nov 03 08:17:16.032091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.965643    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"]
Nov 03 08:17:16.032091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.965781    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.032091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.973529    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e0f576\"/\"openshift-service-ca.crt\""
Nov 03 08:17:16.032091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:15.973978    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e0f576\"/\"kube-root-ca.crt\""
Nov 03 08:17:16.032091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.022345    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:16.044736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.044708    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e0f576\"/\"test-model-e0f576-kserve-self-signed-certs\""
Nov 03 08:17:16.045376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.045357    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e0f576\"/\"default-dockercfg-twmgz\""
Nov 03 08:17:16.158135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.152652    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:16.179735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.173502    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-home\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.179735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.173624    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf911f-3512-4745-a08b-bf6616df4894-tls-certs\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.179735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.173724    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-tmp-dir\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.179735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.173765    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5cz\" (UniqueName: \"kubernetes.io/projected/a9cf911f-3512-4745-a08b-bf6616df4894-kube-api-access-7r5cz\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.179735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.173954    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-model-cache\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.179735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.174088    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-dshm\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.328890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.322725    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-home\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.328890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.323388    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf911f-3512-4745-a08b-bf6616df4894-tls-certs\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.328890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.323657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-tmp-dir\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.328890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.324700    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-tmp-dir\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.328890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.324781    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5cz\" (UniqueName: \"kubernetes.io/projected/a9cf911f-3512-4745-a08b-bf6616df4894-kube-api-access-7r5cz\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.328890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.325259    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-model-cache\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.335192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.335149    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-dshm\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.337387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.337364    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-model-cache\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.338199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.338177    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-home\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.346710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.346653    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-dshm\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.346710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.346670    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5cz\" (UniqueName: \"kubernetes.io/projected/a9cf911f-3512-4745-a08b-bf6616df4894-kube-api-access-7r5cz\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.353444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.353416    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf911f-3512-4745-a08b-bf6616df4894-tls-certs\") pod \"test-model-e0f576-kserve-78fcf7b565-97s6q\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") " pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.390304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.390280    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:16.513730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.513679    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:16.675134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.674241    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:16.801415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:16.785819    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"]
Nov 03 08:17:16.838173 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:17:16.838133    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9cf911f_3512_4745_a08b_bf6616df4894.slice/crio-8dff78e367fd548d4a11bdfb932e58aa3708b30197443cfdaabb9281cac6fe77 WatchSource:0}: Error finding container 8dff78e367fd548d4a11bdfb932e58aa3708b30197443cfdaabb9281cac6fe77: Status 404 returned error can't find the container with id 8dff78e367fd548d4a11bdfb932e58aa3708b30197443cfdaabb9281cac6fe77
Nov 03 08:17:17.132139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:17.132105    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"]
Nov 03 08:17:17.132441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:17.132409    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy" containerID="cri-o://95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec" gracePeriod=30
Nov 03 08:17:17.234974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:17.234882    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q" event={"ID":"a9cf911f-3512-4745-a08b-bf6616df4894","Type":"ContainerStarted","Data":"e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd"}
Nov 03 08:17:17.234974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:17.234960    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q" event={"ID":"a9cf911f-3512-4745-a08b-bf6616df4894","Type":"ContainerStarted","Data":"8dff78e367fd548d4a11bdfb932e58aa3708b30197443cfdaabb9281cac6fe77"}
Nov 03 08:17:17.269263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:17.269196    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q" podStartSLOduration=2.269176811 podStartE2EDuration="2.269176811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:17.260032386 +0000 UTC m=+1386.762867900" watchObservedRunningTime="2026-11-03 08:17:17.269176811 +0000 UTC m=+1386.772012326"
Nov 03 08:17:17.403176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:17.398154    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:17.797668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:17.797630    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:18.859422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:18.859390    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:19.926365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:19.926330    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:20.118891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.118855    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:20.661777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.661753    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:17:20.700270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.700246    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"]
Nov 03 08:17:20.701027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.701014    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy"
Nov 03 08:17:20.701137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.701028    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy"
Nov 03 08:17:20.701215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.701203    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerName="istio-proxy"
Nov 03 08:17:20.705349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.705322    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.720122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.720093    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-podinfo\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.720576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.720315    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-socket\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.720799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.720781    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-data\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.720961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.720943    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-certs\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.721199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.721180    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-token\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.721577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.721557    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istiod-ca-cert\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.721774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.721756    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-credential-socket\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.721905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.721886    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-kube-api-access-5d9r2\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-kube-api-access-5d9r2\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.721979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.721954    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-envoy\") pod \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\" (UID: \"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953\") "
Nov 03 08:17:20.722342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.722320    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7f6f94-openshift-default-dockercfg-9pt96\""
Nov 03 08:17:20.723172 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.722968    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-socket" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:20.723494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.723320    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-credential-socket" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:20.723964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.723727    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istiod-ca-cert" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:20.724305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.724122    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-data" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:20.724524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.724481    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-certs" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:20.725740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.725720    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"]
Nov 03 08:17:20.732791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.731434    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-envoy" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:20.732791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.731451    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-podinfo" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:20.734428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.734407    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-token" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:20.734673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.734650    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-kube-api-access-5d9r2" pod "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" (UID: "e3c73801-07d7-42bb-9f3f-6d3c2c0b6953"). InnerVolumeSpecName "kube-api-access-5d9r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:20.824568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.824540    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5bgw\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-kube-api-access-m5bgw\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.824695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.824645    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-envoy\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.824745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.824724    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-credential-socket\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.824818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.824803    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-data\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.824936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.824923    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-istio-token\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.824981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.824965    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04237f6-0845-4c53-b888-1f0898562832-istiod-ca-cert\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.825017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825001    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-socket\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.825129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825117    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04237f6-0845-4c53-b888-1f0898562832-istio-podinfo\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.825180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825144    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-certs\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.825263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825249    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825278    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825290    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825300    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825309    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825320    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825335    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825346    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9r2\" (UniqueName: \"kubernetes.io/projected/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-kube-api-access-5d9r2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.825361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.825355    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:20.927605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.926741    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5bgw\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-kube-api-access-m5bgw\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.946621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.938095    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-envoy\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.946961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.946929    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-credential-socket\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.947218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.947199    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-data\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.947562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.947535    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-istio-token\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.967180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.967143    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04237f6-0845-4c53-b888-1f0898562832-istiod-ca-cert\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.967581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.967548    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-socket\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.967913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.967891    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04237f6-0845-4c53-b888-1f0898562832-istio-podinfo\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.968098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.968051    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-certs\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.968725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.968699    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-certs\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.968856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.947723    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-credential-socket\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.973426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.948026    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-data\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.973426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.971592    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-socket\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:20.984439 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:20.984409    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04237f6-0845-4c53-b888-1f0898562832-istiod-ca-cert\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:21.020645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.007375    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-envoy\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:21.020645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.008026    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5bgw\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-kube-api-access-m5bgw\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:21.020645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.020588    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04237f6-0845-4c53-b888-1f0898562832-istio-podinfo\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:21.072095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.046590    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-istio-token\") pod \"e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") " pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:21.337106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.337048    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:21.467247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.467208    6430 generic.go:356] "Generic (PLEG): container finished" podID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" containerID="b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a" exitCode=0
Nov 03 08:17:21.467420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.467268    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"
Nov 03 08:17:21.467420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.467275    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" event={"ID":"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953","Type":"ContainerDied","Data":"b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a"}
Nov 03 08:17:21.467420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.467327    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4" event={"ID":"e3c73801-07d7-42bb-9f3f-6d3c2c0b6953","Type":"ContainerDied","Data":"006f1054f4584a560f7bb2231fbb1b04a2ddf9cb85e576c1a6abb83d6bfb9a9b"}
Nov 03 08:17:21.467420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.467359    6430 scope.go:122] "RemoveContainer" containerID="b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a"
Nov 03 08:17:21.482980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.482876    6430 scope.go:122] "RemoveContainer" containerID="b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a"
Nov 03 08:17:21.483463 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:17:21.483389    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a\": container with ID starting with b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a not found: ID does not exist" containerID="b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a"
Nov 03 08:17:21.483463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.483426    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a"} err="failed to get container status \"b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a\": rpc error: code = NotFound desc = could not find container \"b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a\": container with ID starting with b80237774a656944f7eccb60ffaa0d7787dead57a4f37c8b57f2cbc3de0d3d6a not found: ID does not exist"
Nov 03 08:17:21.613135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.599145    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:21.628577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.622128    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"]
Nov 03 08:17:21.631101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.630179    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-rlcl4"]
Nov 03 08:17:21.653845 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:17:21.653814    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04237f6_0845_4c53_b888_1f0898562832.slice/crio-b520f12578635fe7ce79fa6e22b5f51a40b6a75adaaf5bdfeef57cd6bc8d9280 WatchSource:0}: Error finding container b520f12578635fe7ce79fa6e22b5f51a40b6a75adaaf5bdfeef57cd6bc8d9280: Status 404 returned error can't find the container with id b520f12578635fe7ce79fa6e22b5f51a40b6a75adaaf5bdfeef57cd6bc8d9280
Nov 03 08:17:21.654565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.654535    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"]
Nov 03 08:17:21.657505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.657468    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:17:21.657576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.657556    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:17:21.657621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:21.657595    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:17:22.426947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.426924    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:17:22.459090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.459048    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"]
Nov 03 08:17:22.459988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.459964    6430 cpu_manager.go:414] "containerMap: removing container" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy"
Nov 03 08:17:22.459988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.459986    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy"
Nov 03 08:17:22.460297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.460280    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerName="istio-proxy"
Nov 03 08:17:22.464019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.463995    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.472481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.472440    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-99410b-openshift-default-dockercfg-64wgq\""
Nov 03 08:17:22.481007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.480394    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"]
Nov 03 08:17:22.518450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.518415    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" event={"ID":"b04237f6-0845-4c53-b888-1f0898562832","Type":"ContainerStarted","Data":"c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32"}
Nov 03 08:17:22.518723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.518691    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" event={"ID":"b04237f6-0845-4c53-b888-1f0898562832","Type":"ContainerStarted","Data":"b520f12578635fe7ce79fa6e22b5f51a40b6a75adaaf5bdfeef57cd6bc8d9280"}
Nov 03 08:17:22.524480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.524445    6430 generic.go:356] "Generic (PLEG): container finished" podID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" containerID="95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec" exitCode=0
Nov 03 08:17:22.524743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.524727    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"
Nov 03 08:17:22.525697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.525645    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" event={"ID":"907ee690-cf3d-43f1-8ece-6e7491fd8ad6","Type":"ContainerDied","Data":"95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec"}
Nov 03 08:17:22.525859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.525727    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p" event={"ID":"907ee690-cf3d-43f1-8ece-6e7491fd8ad6","Type":"ContainerDied","Data":"2f21fefe95712ed2fc820a30e8eddb254c20b90d5d9cdf4959edfba5058dff57"}
Nov 03 08:17:22.525859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.525760    6430 scope.go:122] "RemoveContainer" containerID="95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec"
Nov 03 08:17:22.541771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.541716    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podStartSLOduration=52.541701429 podStartE2EDuration="52.541701429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:22.538927522 +0000 UTC m=+1392.041763029" watchObservedRunningTime="2026-11-03 08:17:22.541701429 +0000 UTC m=+1392.044536944"
Nov 03 08:17:22.544740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.544712    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-envoy\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.544869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.544789    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-socket\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.544869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.544844    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-certs\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.544987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.544880    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-token\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-token\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.545041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.544993    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-kube-api-access-z7ftv\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-kube-api-access-z7ftv\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.545143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545076    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-podinfo\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.545252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545233    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-data\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.545326 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545293    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-certs" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:22.545326 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545311    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-credential-socket\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.545496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545478    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istiod-ca-cert\") pod \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\" (UID: \"907ee690-cf3d-43f1-8ece-6e7491fd8ad6\") "
Nov 03 08:17:22.545732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545705    6430 scope.go:122] "RemoveContainer" containerID="95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec"
Nov 03 08:17:22.545862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545772    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6471a024-4446-459d-b40b-7f1f4009af97-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.545935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545872    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.545935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545878    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-socket" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:22.546040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.545983    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.546135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546121    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6471a024-4446-459d-b40b-7f1f4009af97-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.546210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546188    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.546267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546219    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lzl\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-kube-api-access-96lzl\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.546320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546273    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.546396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546379    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.546465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546453    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.546714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546695    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.546811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546722    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.546811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546692    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istiod-ca-cert" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:22.546811 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:17:22.546775    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec\": container with ID starting with 95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec not found: ID does not exist" containerID="95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec"
Nov 03 08:17:22.546992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.546807    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec"} err="failed to get container status \"95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec\": rpc error: code = NotFound desc = could not find container \"95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec\": container with ID starting with 95c3eeea290f3115bd95d10982b911d02f4afa972dcf7ce4997f5cf1104498ec not found: ID does not exist"
Nov 03 08:17:22.547141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.547020    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-data" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:22.547218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.547193    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-credential-socket" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:22.548722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.548697    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-podinfo" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:22.548962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.548939    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-envoy" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:22.549096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.548966    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-kube-api-access-z7ftv" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "kube-api-access-z7ftv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:22.549238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.549215    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-token" pod "907ee690-cf3d-43f1-8ece-6e7491fd8ad6" (UID: "907ee690-cf3d-43f1-8ece-6e7491fd8ad6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:22.620980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.620946    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c73801-07d7-42bb-9f3f-6d3c2c0b6953" path="/var/lib/kubelet/pods/e3c73801-07d7-42bb-9f3f-6d3c2c0b6953/volumes"
Nov 03 08:17:22.647696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.647642    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.647822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.647702    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.647977    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6471a024-4446-459d-b40b-7f1f4009af97-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648036    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648100    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648126    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648180    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648270    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6471a024-4446-459d-b40b-7f1f4009af97-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648313    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648331    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lzl\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-kube-api-access-96lzl\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648358    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.648859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.648819    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.649192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649162    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6471a024-4446-459d-b40b-7f1f4009af97-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.649306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649227    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.649306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649247    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.649306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649265    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.649306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649280    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.649306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649295    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.649557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649312    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ftv\" (UniqueName: \"kubernetes.io/projected/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-kube-api-access-z7ftv\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.649557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.649327    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ee690-cf3d-43f1-8ece-6e7491fd8ad6-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:22.651321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.651296    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.651530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.651513    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6471a024-4446-459d-b40b-7f1f4009af97-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.657642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.657585    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lzl\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-kube-api-access-96lzl\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.657733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.657691    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-46r6r\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.696307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.696283    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:22.835728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.829704    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:22.880312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.880264    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"]
Nov 03 08:17:22.886309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:22.886277    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-76k6p"]
Nov 03 08:17:23.170478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.170452    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"]
Nov 03 08:17:23.171267 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:17:23.171241    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6471a024_4446_459d_b40b_7f1f4009af97.slice/crio-9a05d788cc59dbb3963c2863fd3bf4848aef0b3cf6194f6120a06fb3470c389f WatchSource:0}: Error finding container 9a05d788cc59dbb3963c2863fd3bf4848aef0b3cf6194f6120a06fb3470c389f: Status 404 returned error can't find the container with id 9a05d788cc59dbb3963c2863fd3bf4848aef0b3cf6194f6120a06fb3470c389f
Nov 03 08:17:23.174368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.174335    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:17:23.174451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.174421    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:17:23.174489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.174467    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:17:23.351091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.341775    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:23.448510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.448389    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:23.541600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.541565    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" event={"ID":"6471a024-4446-459d-b40b-7f1f4009af97","Type":"ContainerStarted","Data":"1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17"}
Nov 03 08:17:23.541855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.541823    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:23.541979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.541874    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" event={"ID":"6471a024-4446-459d-b40b-7f1f4009af97","Type":"ContainerStarted","Data":"9a05d788cc59dbb3963c2863fd3bf4848aef0b3cf6194f6120a06fb3470c389f"}
Nov 03 08:17:23.542934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.542912    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:17:23.592344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.592299    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podStartSLOduration=5.5922856020000005 podStartE2EDuration="5.592285602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:23.589858739 +0000 UTC m=+1393.092694262" watchObservedRunningTime="2026-11-03 08:17:23.592285602 +0000 UTC m=+1393.095121116"
Nov 03 08:17:23.830368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.830293    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:23.970304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:23.970267    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:24.633075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:24.633033    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907ee690-cf3d-43f1-8ece-6e7491fd8ad6" path="/var/lib/kubelet/pods/907ee690-cf3d-43f1-8ece-6e7491fd8ad6/volumes"
Nov 03 08:17:24.832560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:24.832529    6430 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 08:17:24.832718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:24.832586    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 08:17:25.194319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:25.194275    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:25.668382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:25.668289    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:25.830669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:25.830625    6430 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.830851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:25.830707    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 08:17:25.841640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:25.841611    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:26.391333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:26.391305    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:26.432434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:26.432402    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:26.537087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:26.527625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:26.666187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:26.665584    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:26.690183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:26.670334    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:17:26.843124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:26.843028    6430 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 08:17:26.843124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:26.843118    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 08:17:27.142885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:27.142300    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:27.279330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:27.279300    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:27.864273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:27.864050    6430 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.864273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:27.864174    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 08:17:28.176837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:28.176754    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:28.340803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:28.340769    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:28.435846 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:17:28.435807    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:17:28.517087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:28.509283    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:28.832706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:28.832624    6430 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 08:17:28.832706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:28.832681    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 08:17:29.515086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:29.514163    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:29.694025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:29.693980    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:29.904147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:29.890882    6430 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 08:17:29.908085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:29.904468    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 08:17:29.926139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:29.926104    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:30.107293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.107263    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"]
Nov 03 08:17:30.176631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.174268    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:30.187863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.187771    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.197120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.192870    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"]
Nov 03 08:17:30.197120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.195457    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-616e46\"/\"default-dockercfg-gvhbk\""
Nov 03 08:17:30.197120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.196406    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-616e46\"/\"test-model-616e46-kserve-self-signed-certs\""
Nov 03 08:17:30.248093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.246363    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-616e46\"/\"openshift-service-ca.crt\""
Nov 03 08:17:30.319641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.319602    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:30.323653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.323519    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjfv\" (UniqueName: \"kubernetes.io/projected/7ee9f9ca-9623-4300-9b04-6c3ba857841d-kube-api-access-hcjfv\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.323999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.323980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-model-cache\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.324294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.324276    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tmp-dir\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.324466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.324452    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tls-certs\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.324566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.324552    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-dshm\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.324840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.324823    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-home\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.333349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.333323    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-616e46\"/\"kube-root-ca.crt\""
Nov 03 08:17:30.472368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.472328    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-home\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.498292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.498234    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjfv\" (UniqueName: \"kubernetes.io/projected/7ee9f9ca-9623-4300-9b04-6c3ba857841d-kube-api-access-hcjfv\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.510117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.498696    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-model-cache\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.512103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.512043    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tmp-dir\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.512401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.512377    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tls-certs\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.536250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.536201    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-dshm\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.537259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.474675    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-home\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.537482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.501180    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-model-cache\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.537613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.524439    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tmp-dir\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.937696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.937659    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-dshm\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.938384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.938284    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tls-certs\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.938901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.938827    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjfv\" (UniqueName: \"kubernetes.io/projected/7ee9f9ca-9623-4300-9b04-6c3ba857841d-kube-api-access-hcjfv\") pod \"test-model-616e46-kserve-7787899fd7-jftrb\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") " pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:30.939696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.939629    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:31.006089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:30.980797    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:31.112055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:31.112020    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:31.125392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:31.125179    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:31.166816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:31.166590    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:31.174810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:31.172054    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:31.205211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:31.202369    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:17:31.270092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:31.269900    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:31.686407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:31.686374    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"]
Nov 03 08:17:31.687358 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:17:31.687329    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ee9f9ca_9623_4300_9b04_6c3ba857841d.slice/crio-c4074d727442b2bc2c0be05991d99351ddfb9b1638b3e4c0073441ddd05f3004 WatchSource:0}: Error finding container c4074d727442b2bc2c0be05991d99351ddfb9b1638b3e4c0073441ddd05f3004: Status 404 returned error can't find the container with id c4074d727442b2bc2c0be05991d99351ddfb9b1638b3e4c0073441ddd05f3004
Nov 03 08:17:32.117205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:32.115830    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:32.217176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:32.216701    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" event={"ID":"7ee9f9ca-9623-4300-9b04-6c3ba857841d","Type":"ContainerStarted","Data":"e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377"}
Nov 03 08:17:32.217176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:32.216748    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" event={"ID":"7ee9f9ca-9623-4300-9b04-6c3ba857841d","Type":"ContainerStarted","Data":"c4074d727442b2bc2c0be05991d99351ddfb9b1638b3e4c0073441ddd05f3004"}
Nov 03 08:17:32.240847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:32.240784    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" podStartSLOduration=2.240765299 podStartE2EDuration="2.240765299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:32.238341036 +0000 UTC m=+1401.741176552" watchObservedRunningTime="2026-11-03 08:17:32.240765299 +0000 UTC m=+1401.743600819"
Nov 03 08:17:32.982157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:32.982124    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:33.052656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.052625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:33.459343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.459307    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"]
Nov 03 08:17:33.482509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.482480    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.487268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.486874    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"]
Nov 03 08:17:33.488777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.488753    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b559e\"/\"openshift-service-ca.crt\""
Nov 03 08:17:33.489040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.489025    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b559e\"/\"default-dockercfg-mc56h\""
Nov 03 08:17:33.489185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.488754    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b559e\"/\"kube-root-ca.crt\""
Nov 03 08:17:33.489278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.489224    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b559e\"/\"test-model-9b559e-kserve-self-signed-certs\""
Nov 03 08:17:33.554810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.554779    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:33.590231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.590197    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607fca89-8aa0-4714-820a-393ab37084c0-tls-certs\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.590395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.590368    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-home\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.590453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.590441    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2klgd\" (UniqueName: \"kubernetes.io/projected/607fca89-8aa0-4714-820a-393ab37084c0-kube-api-access-2klgd\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.590514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.590494    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-model-cache\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.590555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.590536    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-dshm\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.590647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.590634    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-tmp-dir\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.698078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.696553    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-home\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.698078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.696757    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2klgd\" (UniqueName: \"kubernetes.io/projected/607fca89-8aa0-4714-820a-393ab37084c0-kube-api-access-2klgd\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.698078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.696821    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-model-cache\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.698078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.696878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-dshm\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.698078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.697036    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-tmp-dir\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.698078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.697322    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607fca89-8aa0-4714-820a-393ab37084c0-tls-certs\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.705133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.698995    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-home\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.705133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.700534    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-model-cache\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.740157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.730055    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-tmp-dir\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.795095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.777944    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2klgd\" (UniqueName: \"kubernetes.io/projected/607fca89-8aa0-4714-820a-393ab37084c0-kube-api-access-2klgd\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.795095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.780216    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-dshm\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.795095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.787761    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607fca89-8aa0-4714-820a-393ab37084c0-tls-certs\") pod \"test-model-9b559e-kserve-7867d7dfbf-q62s7\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") " pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:33.823090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:33.821040    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:34.161509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:34.161478    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:34.211860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:34.211829    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"]
Nov 03 08:17:34.212330 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:17:34.212306    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod607fca89_8aa0_4714_820a_393ab37084c0.slice/crio-d9d207537826f29431bdab7c76766b37e8e6ffe6d3658d01fde200adb47e2d7a WatchSource:0}: Error finding container d9d207537826f29431bdab7c76766b37e8e6ffe6d3658d01fde200adb47e2d7a: Status 404 returned error can't find the container with id d9d207537826f29431bdab7c76766b37e8e6ffe6d3658d01fde200adb47e2d7a
Nov 03 08:17:34.232873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:34.232851    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" event={"ID":"607fca89-8aa0-4714-820a-393ab37084c0","Type":"ContainerStarted","Data":"d9d207537826f29431bdab7c76766b37e8e6ffe6d3658d01fde200adb47e2d7a"}
Nov 03 08:17:35.052208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:35.052170    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:35.242339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:35.242303    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" event={"ID":"607fca89-8aa0-4714-820a-393ab37084c0","Type":"ContainerStarted","Data":"03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666"}
Nov 03 08:17:35.261035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:35.260979    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" podStartSLOduration=2.260965637 podStartE2EDuration="2.260965637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:35.257922339 +0000 UTC m=+1404.760757854" watchObservedRunningTime="2026-11-03 08:17:35.260965637 +0000 UTC m=+1404.763801165"
Nov 03 08:17:35.830089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:35.823473    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:36.073030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:36.072979    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:36.202872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:36.202824    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:36.346087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:36.339484    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:37.179744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:37.179700    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:37.381188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:37.381154    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:37.503286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:37.503254    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:37.864877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:37.864787    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:37.992501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:37.992470    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:38.558517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:38.558484    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:39.288898 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:39.288865    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:39.423294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:39.423258    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:40.072439 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:40.072398    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:40.981809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:40.981771    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:40.994830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:40.994803    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:41.238168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:41.228297    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:41.330957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:41.330928    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:17:42.272109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:42.272081    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:42.899175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:42.899144    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:43.039364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.039334    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:43.174639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.174517    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:43.360203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.360167    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"]
Nov 03 08:17:43.366598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.366574    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.388564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.388511    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b8a152\"/\"openshift-service-ca.crt\""
Nov 03 08:17:43.395247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.395210    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b8a152\"/\"default-dockercfg-8d8ll\""
Nov 03 08:17:43.395427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.395371    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b8a152\"/\"test-model-b8a152-kserve-self-signed-certs\""
Nov 03 08:17:43.395492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.395440    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b8a152\"/\"kube-root-ca.crt\""
Nov 03 08:17:43.404526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.403599    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"]
Nov 03 08:17:43.445246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.445209    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:43.446874 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.446844    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-model-cache\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.447113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.447092    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-home\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.447242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.447148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnsv\" (UniqueName: \"kubernetes.io/projected/31bab6fc-0528-47ae-836e-743075206e82-kube-api-access-smnsv\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.447320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.447308    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-tmp-dir\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.447376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.447361    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bab6fc-0528-47ae-836e-743075206e82-tls-certs\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.447459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.447445    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-dshm\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.565570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.565533    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-model-cache\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.565953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.565930    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-home\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.566280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.566260    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnsv\" (UniqueName: \"kubernetes.io/projected/31bab6fc-0528-47ae-836e-743075206e82-kube-api-access-smnsv\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.566651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.566633    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-tmp-dir\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.566842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.566824    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bab6fc-0528-47ae-836e-743075206e82-tls-certs\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.566983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.566970    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-dshm\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.576090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.572783    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-model-cache\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.576090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.573091    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-home\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.576090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.573347    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-tmp-dir\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.642235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.642205    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-dshm\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.645342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.645314    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnsv\" (UniqueName: \"kubernetes.io/projected/31bab6fc-0528-47ae-836e-743075206e82-kube-api-access-smnsv\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.646079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.646019    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bab6fc-0528-47ae-836e-743075206e82-tls-certs\") pod \"test-model-b8a152-kserve-9965564cc-jjqcl\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") " pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.717614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.717526    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:43.841117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.825480    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:43.944434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:43.943956    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:44.187715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:44.187661    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:44.356086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:44.355981    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:17:44.432727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:44.432695    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"]
Nov 03 08:17:44.433207 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:17:44.433089    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31bab6fc_0528_47ae_836e_743075206e82.slice/crio-898dc453559c2facbc12135f1c0a737ec1e724db2e06f8ae9f62c27f47eec669 WatchSource:0}: Error finding container 898dc453559c2facbc12135f1c0a737ec1e724db2e06f8ae9f62c27f47eec669: Status 404 returned error can't find the container with id 898dc453559c2facbc12135f1c0a737ec1e724db2e06f8ae9f62c27f47eec669
Nov 03 08:17:44.886081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:44.885876    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:45.353777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:45.353735    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" event={"ID":"31bab6fc-0528-47ae-836e-743075206e82","Type":"ContainerStarted","Data":"71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d"}
Nov 03 08:17:45.353981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:45.353789    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" event={"ID":"31bab6fc-0528-47ae-836e-743075206e82","Type":"ContainerStarted","Data":"898dc453559c2facbc12135f1c0a737ec1e724db2e06f8ae9f62c27f47eec669"}
Nov 03 08:17:45.372583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:45.372536    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" podStartSLOduration=2.372523372 podStartE2EDuration="2.372523372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:45.368569103 +0000 UTC m=+1414.871404619" watchObservedRunningTime="2026-11-03 08:17:45.372523372 +0000 UTC m=+1414.875358886"
Nov 03 08:17:45.718201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:45.718175    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:45.886844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:45.886804    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:46.673271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:46.673238    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:47.458544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:47.458505    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:48.430552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:48.430520    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:49.300334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:49.300293    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:50.543026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:50.542992    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:51.565823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:51.565789    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:51.770717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:51.770662    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:51.990694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:51.990662    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:52.612368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:52.612322    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:53.429933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:53.429892    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:53.718845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:53.718819    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:53.731447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:53.731423    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:54.147012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:54.146939    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:54.284090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:54.279902    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:54.521111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:54.521086    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:17:54.998468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:54.998412    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:55.998201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:55.998171    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:56.867260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:56.867225    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:57.691009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:57.690975    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:57.812730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:57.812607    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:58.381514 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:17:58.381476    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:17:58.454546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:58.454512    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:17:59.231938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:17:59.231900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:00.034613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:00.034579    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:00.838261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:00.838230    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:01.623488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:01.623456    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:03.194767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:03.194728    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:03.445235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:03.445135    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:04.145773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:04.145740    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:04.878803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:04.878754    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:05.809397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:05.809363    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:06.531455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:06.531425    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:07.344758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:07.344721    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:08.123936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:08.123901    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:08.844488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:08.844456    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:09.554915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:09.554886    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:10.276238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:10.276199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:10.956776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:10.956745    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:11.708744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:11.708714    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:12.448662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:12.448628    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:13.141742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:13.141707    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:13.650797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:13.650751    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"]
Nov 03 08:18:13.651340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:13.651011    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp" podUID="8461f6e4-df91-4a0b-9cc4-0bda70a26a76" containerName="maas-api" containerID="cri-o://b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2" gracePeriod=30
Nov 03 08:18:13.680613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:13.680572    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:13.888531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:13.888479    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:14.083034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.082998    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:14.083840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.083405    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:18:14.165081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.165015    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"]
Nov 03 08:18:14.170734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.170697    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8461f6e4-df91-4a0b-9cc4-0bda70a26a76" containerName="maas-api"
Nov 03 08:18:14.170966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.170947    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8461f6e4-df91-4a0b-9cc4-0bda70a26a76" containerName="maas-api"
Nov 03 08:18:14.171432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.171408    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8461f6e4-df91-4a0b-9cc4-0bda70a26a76" containerName="maas-api"
Nov 03 08:18:14.195906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.195876    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.205467 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.205437    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-616e46-openshift-default-dockercfg-mqp6b\""
Nov 03 08:18:14.207831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.207803    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:14.222776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.222741    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-metrics-certs\") pod \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") "
Nov 03 08:18:14.231022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.230992    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-openshift-service-ca\") pod \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") "
Nov 03 08:18:14.238996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.238965    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-kube-api-access-jqcm2\" (UniqueName: \"kubernetes.io/projected/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-kube-api-access-jqcm2\") pod \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") "
Nov 03 08:18:14.243597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.243568    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-maas-api-tls\") pod \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\" (UID: \"8461f6e4-df91-4a0b-9cc4-0bda70a26a76\") "
Nov 03 08:18:14.254083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.244137    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.255465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.255436    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.255760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.255739    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.255991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.255975    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.282610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.282576    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcqb\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-kube-api-access-8lcqb\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.283845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.283816    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.284047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.284028    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d001284-d2f0-4e54-af47-26b578dc8596-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.284274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.284255    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.284398 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.284384    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d001284-d2f0-4e54-af47-26b578dc8596-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.284704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.237478    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-openshift-service-ca" pod "8461f6e4-df91-4a0b-9cc4-0bda70a26a76" (UID: "8461f6e4-df91-4a0b-9cc4-0bda70a26a76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:18:14.285189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.276271    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-kube-api-access-jqcm2" pod "8461f6e4-df91-4a0b-9cc4-0bda70a26a76" (UID: "8461f6e4-df91-4a0b-9cc4-0bda70a26a76"). InnerVolumeSpecName "kube-api-access-jqcm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:18:14.285306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.282266    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-metrics-certs" pod "8461f6e4-df91-4a0b-9cc4-0bda70a26a76" (UID: "8461f6e4-df91-4a0b-9cc4-0bda70a26a76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:18:14.305276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.305211    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-maas-api-tls" pod "8461f6e4-df91-4a0b-9cc4-0bda70a26a76" (UID: "8461f6e4-df91-4a0b-9cc4-0bda70a26a76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:18:14.423083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.420742    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcqb\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-kube-api-access-8lcqb\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.423083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.420832    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.423083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.420867    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d001284-d2f0-4e54-af47-26b578dc8596-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.423083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.420934    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.423083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.420965    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d001284-d2f0-4e54-af47-26b578dc8596-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480206    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480309    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480428    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480562    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480914    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqcm2\" (UniqueName: \"kubernetes.io/projected/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-kube-api-access-jqcm2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480945    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480962    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:14.484165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.480977    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8461f6e4-df91-4a0b-9cc4-0bda70a26a76-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:14.508866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.495576    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.583782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.583745    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:14.585725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.585695    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.586351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.586326    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d001284-d2f0-4e54-af47-26b578dc8596-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.589082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.589043    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.589342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.589322    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.589791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.589553    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.599532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.599504    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.600730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.600710    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"]
Nov 03 08:18:14.604199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.604135    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcqb\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-kube-api-access-8lcqb\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.605150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.605120    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d001284-d2f0-4e54-af47-26b578dc8596-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:14.711935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.707678    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:14.781144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.780034    6430 generic.go:356] "Generic (PLEG): container finished" podID="8461f6e4-df91-4a0b-9cc4-0bda70a26a76" containerID="b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2" exitCode=0
Nov 03 08:18:14.781144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.780180    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"
Nov 03 08:18:14.781379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.780201    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp" event={"ID":"8461f6e4-df91-4a0b-9cc4-0bda70a26a76","Type":"ContainerDied","Data":"b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2"}
Nov 03 08:18:14.781379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.781365    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp" event={"ID":"8461f6e4-df91-4a0b-9cc4-0bda70a26a76","Type":"ContainerDied","Data":"bad353ba305b1e263635b501a12fb3e8daf69e5fddf823ca0135f1dc2cb22d3d"}
Nov 03 08:18:14.781491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.781414    6430 scope.go:122] "RemoveContainer" containerID="b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2"
Nov 03 08:18:14.811810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.811783    6430 scope.go:122] "RemoveContainer" containerID="b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2"
Nov 03 08:18:14.812358 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:18:14.812330    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2\": container with ID starting with b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2 not found: ID does not exist" containerID="b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2"
Nov 03 08:18:14.812464 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.812380    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2"} err="failed to get container status \"b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2\": rpc error: code = NotFound desc = could not find container \"b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2\": container with ID starting with b6c28fe4c1e5167f0e47004da877f1885fa3492992bd652b41ada4a350bed5c2 not found: ID does not exist"
Nov 03 08:18:14.816159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.816129    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"]
Nov 03 08:18:14.822144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.822123    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-950e06-5dbff4b58-8r9fp"]
Nov 03 08:18:14.884374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:14.884354    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:15.173505 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:18:15.173473    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d001284_d2f0_4e54_af47_26b578dc8596.slice/crio-be4f70fa90e772cccf917e55d2c3b4e09cd6bd821662eee58fdbc741213bd378 WatchSource:0}: Error finding container be4f70fa90e772cccf917e55d2c3b4e09cd6bd821662eee58fdbc741213bd378: Status 404 returned error can't find the container with id be4f70fa90e772cccf917e55d2c3b4e09cd6bd821662eee58fdbc741213bd378
Nov 03 08:18:15.173628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.173606    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"]
Nov 03 08:18:15.176931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.176891    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:18:15.177037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.176998    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:18:15.177138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.177039    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:18:15.417455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.417417    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:15.790775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.790741    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" event={"ID":"6d001284-d2f0-4e54-af47-26b578dc8596","Type":"ContainerStarted","Data":"26e36941f422b7e1e8334db4d17477f201af1e587f7a2fc975aa655576a89b2f"}
Nov 03 08:18:15.791243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.790795    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" event={"ID":"6d001284-d2f0-4e54-af47-26b578dc8596","Type":"ContainerStarted","Data":"be4f70fa90e772cccf917e55d2c3b4e09cd6bd821662eee58fdbc741213bd378"}
Nov 03 08:18:15.825616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.825563    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" podStartSLOduration=29.825548536 podStartE2EDuration="29.825548536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:18:15.822600268 +0000 UTC m=+1445.325435785" watchObservedRunningTime="2026-11-03 08:18:15.825548536 +0000 UTC m=+1445.328384049"
Nov 03 08:18:15.885411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:15.885377    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:16.141098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:16.140985    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:16.633434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:16.633405    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8461f6e4-df91-4a0b-9cc4-0bda70a26a76" path="/var/lib/kubelet/pods/8461f6e4-df91-4a0b-9cc4-0bda70a26a76/volumes"
Nov 03 08:18:16.862389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:16.862358    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:16.915141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:16.909313    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 08:18:16.915141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:16.909383    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 08:18:17.561769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:17.561739    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:17.885800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:17.885706    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 08:18:17.885800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:17.885758    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 08:18:18.350797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:18.350760    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:18.885344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:18.885311    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 08:18:18.885532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:18.885373    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 08:18:19.062665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:19.062634    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:19.762038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:19.761952    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:19.896029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:19.895987    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 08:18:19.896439 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:19.896394    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 08:18:20.585663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:20.585630    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:20.885429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:20.885348    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 08:18:20.885429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:20.885407    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 08:18:21.222604 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:21.222566    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:21.340281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:21.340253    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:21.480195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:21.477309    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:21.690161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:21.621961    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:22.096440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:22.096407    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:22.098446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:22.097808    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:22.099780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:22.099759    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:18:22.791164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:22.789942    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:23.457441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:23.457410    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:24.147316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:24.147279    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:24.916654 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:24.916625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:25.605425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:25.605395    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:26.271680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:26.271653    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:27.140434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:27.140397    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:27.937409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:27.937372    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:28.378213 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:18:28.378131    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:18:28.793834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:28.793798    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:29.644826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:29.644795    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:30.392824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:30.392788    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:31.309998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:31.309970    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:32.248946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:32.248866    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:32.412721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:32.412682    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"]
Nov 03 08:18:32.428305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:32.428272    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"]
Nov 03 08:18:32.543207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:32.543109    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:32.653208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:32.653178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:33.504131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:33.504091    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:34.293741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:34.293707    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:35.054145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:35.054110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:35.928141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:35.928108    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:36.618958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:36.618923    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:37.361043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.361007    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:37.416088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.415142    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr" podUID="c075e012-470f-4bfd-bc4c-16dd635cb65f" containerName="payload-processing" containerID="cri-o://5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4" gracePeriod=25
Nov 03 08:18:37.441174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.433282    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47" podUID="9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" containerName="payload-pre-processing" containerID="cri-o://6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24" gracePeriod=25
Nov 03 08:18:37.759451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.759421    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:18:37.799487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.799460    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:18:37.844806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.844769    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c075e012-470f-4bfd-bc4c-16dd635cb65f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c075e012-470f-4bfd-bc4c-16dd635cb65f-plugins-config-volume\") pod \"c075e012-470f-4bfd-bc4c-16dd635cb65f\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") "
Nov 03 08:18:37.845005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.844944    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c075e012-470f-4bfd-bc4c-16dd635cb65f-kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/c075e012-470f-4bfd-bc4c-16dd635cb65f-kube-api-access-wtw7q\") pod \"c075e012-470f-4bfd-bc4c-16dd635cb65f\" (UID: \"c075e012-470f-4bfd-bc4c-16dd635cb65f\") "
Nov 03 08:18:37.845168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.845142    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c075e012-470f-4bfd-bc4c-16dd635cb65f-plugins-config-volume" pod "c075e012-470f-4bfd-bc4c-16dd635cb65f" (UID: "c075e012-470f-4bfd-bc4c-16dd635cb65f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:18:37.845942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.845914    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c075e012-470f-4bfd-bc4c-16dd635cb65f-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:37.848506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.848473    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c075e012-470f-4bfd-bc4c-16dd635cb65f-kube-api-access-wtw7q" pod "c075e012-470f-4bfd-bc4c-16dd635cb65f" (UID: "c075e012-470f-4bfd-bc4c-16dd635cb65f"). InnerVolumeSpecName "kube-api-access-wtw7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:18:37.947173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.947138    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-kube-api-access-9cjq8\" (UniqueName: \"kubernetes.io/projected/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-kube-api-access-9cjq8\") pod \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") "
Nov 03 08:18:37.947370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.947194    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-plugins-config-volume\") pod \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\" (UID: \"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e\") "
Nov 03 08:18:37.947705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.947678    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-plugins-config-volume" pod "9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" (UID: "9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:18:37.947990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.947970    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtw7q\" (UniqueName: \"kubernetes.io/projected/c075e012-470f-4bfd-bc4c-16dd635cb65f-kube-api-access-wtw7q\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:37.947990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.947989    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:37.949587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:37.949566    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-kube-api-access-9cjq8" pod "9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" (UID: "9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e"). InnerVolumeSpecName "kube-api-access-9cjq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:18:38.050003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.049970    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjq8\" (UniqueName: \"kubernetes.io/projected/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e-kube-api-access-9cjq8\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:38.055655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.055625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.217515    6430 generic.go:356] "Generic (PLEG): container finished" podID="9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" containerID="6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24" exitCode=0
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.217600    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.217658    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47" event={"ID":"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e","Type":"ContainerDied","Data":"6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24"}
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.217699    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47" event={"ID":"9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e","Type":"ContainerDied","Data":"324922b351661db4f9554ca3b9fd3d3910adc60bf98355950d0ef677baf77cad"}
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.217732    6430 scope.go:122] "RemoveContainer" containerID="6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24"
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.219577    6430 generic.go:356] "Generic (PLEG): container finished" podID="c075e012-470f-4bfd-bc4c-16dd635cb65f" containerID="5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4" exitCode=0
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.219621    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr" event={"ID":"c075e012-470f-4bfd-bc4c-16dd635cb65f","Type":"ContainerDied","Data":"5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4"}
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.219673    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr" event={"ID":"c075e012-470f-4bfd-bc4c-16dd635cb65f","Type":"ContainerDied","Data":"dc3e1de091b4053250a3532d1e2a538a4a1c9e69c0142d74861fd29e66f721ee"}
Nov 03 08:18:38.222166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.219755    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"
Nov 03 08:18:38.233305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.233217    6430 scope.go:122] "RemoveContainer" containerID="6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24"
Nov 03 08:18:38.233539 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:18:38.233514    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24\": container with ID starting with 6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24 not found: ID does not exist" containerID="6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24"
Nov 03 08:18:38.233661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.233551    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24"} err="failed to get container status \"6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24\": rpc error: code = NotFound desc = could not find container \"6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24\": container with ID starting with 6908e6548609da5db84e1fae7b8e9dc9bb0e8eb716c1cc47063644e745e85a24 not found: ID does not exist"
Nov 03 08:18:38.233661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.233575    6430 scope.go:122] "RemoveContainer" containerID="5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4"
Nov 03 08:18:38.244370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.244346    6430 scope.go:122] "RemoveContainer" containerID="5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4"
Nov 03 08:18:38.244717 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:18:38.244651    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4\": container with ID starting with 5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4 not found: ID does not exist" containerID="5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4"
Nov 03 08:18:38.244717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.244676    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4"} err="failed to get container status \"5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4\": rpc error: code = NotFound desc = could not find container \"5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4\": container with ID starting with 5a6291a438c9a87295c6786ddc6dfcc947a09f75ae4063fe8d94ba6fb56156e4 not found: ID does not exist"
Nov 03 08:18:38.250141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.250116    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"]
Nov 03 08:18:38.254687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.254661    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-99410b-55f5d5b55f-kgb47"]
Nov 03 08:18:38.261712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.261686    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"]
Nov 03 08:18:38.267219 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.267196    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-99410b-5578995bc9-9g2vr"]
Nov 03 08:18:38.620507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.620435    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" path="/var/lib/kubelet/pods/9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e/volumes"
Nov 03 08:18:38.620955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.620940    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c075e012-470f-4bfd-bc4c-16dd635cb65f" path="/var/lib/kubelet/pods/c075e012-470f-4bfd-bc4c-16dd635cb65f/volumes"
Nov 03 08:18:38.761874 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:38.761840    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:39.854422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:39.854392    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:41.308086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:41.301922    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:41.883435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:41.883392    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:41.901213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:41.901172    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:41.993085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:41.991768    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" podUID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.148:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.038204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.038157    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.058465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.058408    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.213150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.201152    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:18:42.293083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.290152    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" podUID="c62a3394-de42-4cda-b192-8157f006be53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.344426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.344363    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" podUID="607fca89-8aa0-4714-820a-393ab37084c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.600135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.599353    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 08:18:42.600135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.599815    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" podUID="31bab6fc-0528-47ae-836e-743075206e82" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": context deadline exceeded"
Nov 03 08:18:42.784132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:42.667691    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.073915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.073870    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 08:18:43.074664 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.074614    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.146086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.128094    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:18:43.146086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.128159    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:18:43.187742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.157955    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:18:43.204096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.198535    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk" podUID="10f80321-1867-4c5d-8462-24a3a17c4831" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.204096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.201387    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.204096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.203016    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:18:43.205319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.205149    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:18:43.244652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.244621    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:18:43.873742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:43.873651    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:44.872413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:44.872376    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:45.866486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:45.866451    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:46.769123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:46.769053    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:47.982489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:47.982455    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:48.872174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:48.872139    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:49.750737 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:49.750706    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:50.652468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:50.652433    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:51.421592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:51.421558    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:52.235086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:52.231538    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:53.057630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:53.057598    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:53.181109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:53.181048    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:53.916545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:53.916515    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:54.092243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:54.092210    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:54.875412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:54.875382    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:55.653033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:55.652998    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:56.411296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:56.411265    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:57.268039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:57.268005    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:58.069428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:58.069396    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:58.378293 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:18:58.378206    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:18:58.854940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:58.854903    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:18:59.783116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:18:59.783084    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:00.549757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:00.549723    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:01.275389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:01.275355    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:02.347151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:02.347115    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:02.947012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:02.946966    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:03.859022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:03.858988    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:04.552841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:04.552771    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:05.362056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:05.362025    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:06.153552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:06.153494    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:06.793208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:06.793174    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"]
Nov 03 08:19:06.793987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:06.793952    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r" podUID="b66d4820-51da-4287-9ef0-3d27b30c63d3" containerName="maas-api" containerID="cri-o://afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81" gracePeriod=30
Nov 03 08:19:06.854533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:06.854502    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:07.025550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.024859    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:07.281086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.257959    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:07.561095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.557616    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:07.735087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.732441    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:19:07.742824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.742795    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:07.756451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.756418    6430 generic.go:356] "Generic (PLEG): container finished" podID="b66d4820-51da-4287-9ef0-3d27b30c63d3" containerID="afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81" exitCode=0
Nov 03 08:19:07.756644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.756501    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r" event={"ID":"b66d4820-51da-4287-9ef0-3d27b30c63d3","Type":"ContainerDied","Data":"afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81"}
Nov 03 08:19:07.756644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.756578    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r" event={"ID":"b66d4820-51da-4287-9ef0-3d27b30c63d3","Type":"ContainerDied","Data":"be691e199b00f1cdd546e712ba6dc7cc8764a48560eabcb01c0ba8835ec6b217"}
Nov 03 08:19:07.756644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.756601    6430 scope.go:122] "RemoveContainer" containerID="afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81"
Nov 03 08:19:07.756950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.756928    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"
Nov 03 08:19:07.773780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.770863    6430 scope.go:122] "RemoveContainer" containerID="afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81"
Nov 03 08:19:07.774532 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:19:07.774502    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81\": container with ID starting with afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81 not found: ID does not exist" containerID="afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81"
Nov 03 08:19:07.774648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.774544    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81"} err="failed to get container status \"afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81\": rpc error: code = NotFound desc = could not find container \"afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81\": container with ID starting with afd90b3cdeee06d4d2e6f8fab6c5c8eae6dec1f09f2b1654d611d50a21050f81 not found: ID does not exist"
Nov 03 08:19:07.776166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.776135    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"]
Nov 03 08:19:07.777281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777218    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c075e012-470f-4bfd-bc4c-16dd635cb65f" containerName="payload-processing"
Nov 03 08:19:07.777281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777250    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c075e012-470f-4bfd-bc4c-16dd635cb65f" containerName="payload-processing"
Nov 03 08:19:07.777435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777287    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" containerName="payload-pre-processing"
Nov 03 08:19:07.777435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777293    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" containerName="payload-pre-processing"
Nov 03 08:19:07.777435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777336    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b66d4820-51da-4287-9ef0-3d27b30c63d3" containerName="maas-api"
Nov 03 08:19:07.777435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777345    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66d4820-51da-4287-9ef0-3d27b30c63d3" containerName="maas-api"
Nov 03 08:19:07.777666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777498    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c075e012-470f-4bfd-bc4c-16dd635cb65f" containerName="payload-processing"
Nov 03 08:19:07.777666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777512    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e54c51d-3bfe-44dc-a4b5-7b1ac2315b5e" containerName="payload-pre-processing"
Nov 03 08:19:07.777666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.777525    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66d4820-51da-4287-9ef0-3d27b30c63d3" containerName="maas-api"
Nov 03 08:19:07.782822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.782807    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.790877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.790850    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"]
Nov 03 08:19:07.816762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.816733    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66d4820-51da-4287-9ef0-3d27b30c63d3-kube-api-access-wh47t\" (UniqueName: \"kubernetes.io/projected/b66d4820-51da-4287-9ef0-3d27b30c63d3-kube-api-access-wh47t\") pod \"b66d4820-51da-4287-9ef0-3d27b30c63d3\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") "
Nov 03 08:19:07.817140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.816858    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-maas-api-tls\") pod \"b66d4820-51da-4287-9ef0-3d27b30c63d3\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") "
Nov 03 08:19:07.817140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.816933    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66d4820-51da-4287-9ef0-3d27b30c63d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66d4820-51da-4287-9ef0-3d27b30c63d3-openshift-service-ca\") pod \"b66d4820-51da-4287-9ef0-3d27b30c63d3\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") "
Nov 03 08:19:07.817140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.816973    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-metrics-certs\") pod \"b66d4820-51da-4287-9ef0-3d27b30c63d3\" (UID: \"b66d4820-51da-4287-9ef0-3d27b30c63d3\") "
Nov 03 08:19:07.817377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.817348    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66d4820-51da-4287-9ef0-3d27b30c63d3-openshift-service-ca" pod "b66d4820-51da-4287-9ef0-3d27b30c63d3" (UID: "b66d4820-51da-4287-9ef0-3d27b30c63d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:19:07.817864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.817839    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66d4820-51da-4287-9ef0-3d27b30c63d3-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:07.819266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.819243    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66d4820-51da-4287-9ef0-3d27b30c63d3-kube-api-access-wh47t" pod "b66d4820-51da-4287-9ef0-3d27b30c63d3" (UID: "b66d4820-51da-4287-9ef0-3d27b30c63d3"). InnerVolumeSpecName "kube-api-access-wh47t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:19:07.819451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.819428    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-metrics-certs" pod "b66d4820-51da-4287-9ef0-3d27b30c63d3" (UID: "b66d4820-51da-4287-9ef0-3d27b30c63d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:19:07.819547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.819471    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-maas-api-tls" pod "b66d4820-51da-4287-9ef0-3d27b30c63d3" (UID: "b66d4820-51da-4287-9ef0-3d27b30c63d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:19:07.919701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.919666    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wpc\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-kube-api-access-h6wpc\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.919871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.919715    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.919871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.919742    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.919871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.919856    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.920013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920000    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.920173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920154    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.920216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920202    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.920264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920250    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.920311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920299    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:07.920421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920407    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh47t\" (UniqueName: \"kubernetes.io/projected/b66d4820-51da-4287-9ef0-3d27b30c63d3-kube-api-access-wh47t\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:07.920459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920426    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:07.920459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:07.920438    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66d4820-51da-4287-9ef0-3d27b30c63d3-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:08.021940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.021900    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.021952    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.021978    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022015    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022111    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wpc\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-kube-api-access-h6wpc\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022145    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022161    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022253    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022389    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022418    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022517    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022630    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.022819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.022800    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.023242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.023225    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.024641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.024622    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.024994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.024976    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.030231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.030176    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.030350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.030321    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wpc\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-kube-api-access-h6wpc\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.084192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.084160    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"]
Nov 03 08:19:08.091926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.091896    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r"]
Nov 03 08:19:08.103501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.103475    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:08.146709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.146665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:08.264242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.264196    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:08.419150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.415618    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:08.452658 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.447352    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"]
Nov 03 08:19:08.477300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.477254    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:19:08.477499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.477350    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:19:08.477499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.477395    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:19:08.622365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.622335    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66d4820-51da-4287-9ef0-3d27b30c63d3" path="/var/lib/kubelet/pods/b66d4820-51da-4287-9ef0-3d27b30c63d3/volumes"
Nov 03 08:19:08.832152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.832095    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" event={"ID":"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6","Type":"ContainerStarted","Data":"d1b62c7e3293ff58ec82c11b61d309ef11dd6d5bdf98138ec9c515b3e57835cb"}
Nov 03 08:19:08.832633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.832168    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" event={"ID":"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6","Type":"ContainerStarted","Data":"b9835b764811a4998db4be89a17d4ad6c617115fff17dd230980352dd3c7c59a"}
Nov 03 08:19:08.902095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:08.901337    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podStartSLOduration=52.901314872 podStartE2EDuration="52.901314872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:19:08.871390741 +0000 UTC m=+1498.374226256" watchObservedRunningTime="2026-11-03 08:19:08.901314872 +0000 UTC m=+1498.404150390"
Nov 03 08:19:09.104492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:09.104372    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:09.105774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:09.105738    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 08:19:09.105947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:09.105797    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 08:19:09.255719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:09.255686    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:10.022089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:10.021562    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:10.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:10.109439    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 08:19:10.127085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:10.109499    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 08:19:10.141333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:10.140306    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99410b-b86c85857-q8p8r" podUID="b66d4820-51da-4287-9ef0-3d27b30c63d3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:19:10.750516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:10.750483    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:11.113140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:11.107365    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 08:19:11.113140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:11.107420    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 08:19:11.490455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:11.484610    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:12.120141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:12.116253    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 08:19:12.120141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:12.116314    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 08:19:12.248177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:12.248145    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:12.968644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:12.968610    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:13.120089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:13.117486    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 08:19:13.120089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:13.117565    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 08:19:13.648708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:13.648674    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:13.762008 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:13.761775    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:14.106090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.105516    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 08:19:14.106090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.105582    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 08:19:14.353447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.353415    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:14.471085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.470665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:14.625088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.622250    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:14.671434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.671288    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"]
Nov 03 08:19:14.671631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.671517    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" containerID="cri-o://1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17" gracePeriod=30
Nov 03 08:19:14.804254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:14.791684    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:15.109149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:15.106986    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 08:19:15.109149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:15.107055    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 08:19:15.481957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:15.481906    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:15.618153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:15.607929    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:15.924382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:15.924262    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:16.105172 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:16.097657    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:16.198086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:16.188898    6430 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:19:16.198086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:16.188963    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:19:16.302087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:16.298360    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:16.302087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:16.300146    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:16.314090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:16.308601    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:19:16.682900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:16.682843    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:17.461866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:17.461823    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:18.138918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:18.138888    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:18.730045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:18.729993    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:18.937087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:18.926687    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:19.611998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:19.611968    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:19.972543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:19.972522    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:19:20.004371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.004343    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"]
Nov 03 08:19:20.005221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.005204    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy"
Nov 03 08:19:20.005287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.005222    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy"
Nov 03 08:19:20.005401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.005392    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6471a024-4446-459d-b40b-7f1f4009af97" containerName="istio-proxy"
Nov 03 08:19:20.033874 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.033844    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-istio-token\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-istio-token\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.033888    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-socket\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.033945    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6471a024-4446-459d-b40b-7f1f4009af97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6471a024-4446-459d-b40b-7f1f4009af97-istio-podinfo\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.033989    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-credential-socket\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034193    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-kube-api-access-96lzl\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-kube-api-access-96lzl\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034258    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-data\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034296    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-socket" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.034400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034346    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-envoy\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034433    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6471a024-4446-459d-b40b-7f1f4009af97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6471a024-4446-459d-b40b-7f1f4009af97-istiod-ca-cert\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034466    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-certs\") pod \"6471a024-4446-459d-b40b-7f1f4009af97\" (UID: \"6471a024-4446-459d-b40b-7f1f4009af97\") "
Nov 03 08:19:20.034675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034650    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-credential-socket" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.034960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.034932    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"]
Nov 03 08:19:20.035099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.035079    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.035609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.035204    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-data" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.035609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.035353    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.035609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.035365    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.035609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.035374    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.036261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.036225    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-certs" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.036343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.036293    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6471a024-4446-459d-b40b-7f1f4009af97-istiod-ca-cert" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:19:20.037800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.037773    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-envoy" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.038935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.038910    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-istio-token" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:19:20.039045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.038950    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-kube-api-access-96lzl" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "kube-api-access-96lzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:19:20.040048 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.040029    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a66d5c-serving-cert\""
Nov 03 08:19:20.046529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.046506    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6471a024-4446-459d-b40b-7f1f4009af97-istio-podinfo" pod "6471a024-4446-459d-b40b-7f1f4009af97" (UID: "6471a024-4446-459d-b40b-7f1f4009af97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:19:20.080624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.080597    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:20.136515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.136473    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-maas-api-tls\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.136716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.136678    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94702209-a919-4445-9577-cd5739032003-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.137245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.136794    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-metrics-certs\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.137245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.136964    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5rn\" (UniqueName: \"kubernetes.io/projected/94702209-a919-4445-9577-cd5739032003-kube-api-access-7p5rn\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.137245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.137195    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.137245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.137216    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6471a024-4446-459d-b40b-7f1f4009af97-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.137245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.137232    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6471a024-4446-459d-b40b-7f1f4009af97-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.137245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.137247    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.137625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.137262    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6471a024-4446-459d-b40b-7f1f4009af97-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.137625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.137277    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lzl\" (UniqueName: \"kubernetes.io/projected/6471a024-4446-459d-b40b-7f1f4009af97-kube-api-access-96lzl\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.146828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.146798    6430 generic.go:356] "Generic (PLEG): container finished" podID="6471a024-4446-459d-b40b-7f1f4009af97" containerID="1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17" exitCode=0
Nov 03 08:19:20.146963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.146861    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"
Nov 03 08:19:20.146963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.146906    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" event={"ID":"6471a024-4446-459d-b40b-7f1f4009af97","Type":"ContainerDied","Data":"1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17"}
Nov 03 08:19:20.147151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.146970    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r" event={"ID":"6471a024-4446-459d-b40b-7f1f4009af97","Type":"ContainerDied","Data":"9a05d788cc59dbb3963c2863fd3bf4848aef0b3cf6194f6120a06fb3470c389f"}
Nov 03 08:19:20.147151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.147000    6430 scope.go:122] "RemoveContainer" containerID="1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17"
Nov 03 08:19:20.174951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.174883    6430 scope.go:122] "RemoveContainer" containerID="1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17"
Nov 03 08:19:20.177410 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:19:20.176119    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17\": container with ID starting with 1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17 not found: ID does not exist" containerID="1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17"
Nov 03 08:19:20.177410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.176166    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17"} err="failed to get container status \"1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17\": rpc error: code = NotFound desc = could not find container \"1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17\": container with ID starting with 1654accdfdaf60de63c1ae985adc93bad59a97a9493e3c908c68dc0590ddbb17 not found: ID does not exist"
Nov 03 08:19:20.182264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.182243    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"]
Nov 03 08:19:20.189316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.189293    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-46r6r"]
Nov 03 08:19:20.207970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.207916    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:20.239158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.239133    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94702209-a919-4445-9577-cd5739032003-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.239272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.239203    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-metrics-certs\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.239338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.239307    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5rn\" (UniqueName: \"kubernetes.io/projected/94702209-a919-4445-9577-cd5739032003-kube-api-access-7p5rn\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.239454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.239440    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-maas-api-tls\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.239840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.239819    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94702209-a919-4445-9577-cd5739032003-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.241928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.241905    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-metrics-certs\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.242006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.241910    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-maas-api-tls\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.246660 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.246633    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5rn\" (UniqueName: \"kubernetes.io/projected/94702209-a919-4445-9577-cd5739032003-kube-api-access-7p5rn\") pod \"maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb\" (UID: \"94702209-a919-4445-9577-cd5739032003\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.382773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.382745    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:20.443287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.443249    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:20.624376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.624347    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6471a024-4446-459d-b40b-7f1f4009af97" path="/var/lib/kubelet/pods/6471a024-4446-459d-b40b-7f1f4009af97/volumes"
Nov 03 08:19:20.671688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:20.671656    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"]
Nov 03 08:19:20.672659 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:19:20.672633    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94702209_a919_4445_9577_cd5739032003.slice/crio-f289e25e71613f336aa8f27ec137a357b9ec30c3135f6bf06bb1c91e32517ab6 WatchSource:0}: Error finding container f289e25e71613f336aa8f27ec137a357b9ec30c3135f6bf06bb1c91e32517ab6: Status 404 returned error can't find the container with id f289e25e71613f336aa8f27ec137a357b9ec30c3135f6bf06bb1c91e32517ab6
Nov 03 08:19:21.156674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:21.156639    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb" event={"ID":"94702209-a919-4445-9577-cd5739032003","Type":"ContainerStarted","Data":"f289e25e71613f336aa8f27ec137a357b9ec30c3135f6bf06bb1c91e32517ab6"}
Nov 03 08:19:21.326827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:21.325632    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:22.171639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:22.171599    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb" event={"ID":"94702209-a919-4445-9577-cd5739032003","Type":"ContainerStarted","Data":"bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d"}
Nov 03 08:19:22.172119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:22.171963    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:22.191787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:22.191744    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb" podStartSLOduration=90.625242043 podStartE2EDuration="1m31.191729228s" totalImagesPullingTime="566.487185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:19:22.187870421 +0000 UTC m=+1511.690705935" watchObservedRunningTime="2026-11-03 08:19:22.191729228 +0000 UTC m=+1511.694565127"
Nov 03 08:19:22.227881 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:22.227839    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:23.341029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:23.340980    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:24.075521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:24.075485    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:24.189927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:24.189873    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:24.410581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:24.410505    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:25.475600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.475570    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:25.516179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.516150    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"]
Nov 03 08:19:25.520913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.520894    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:25.529546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.529516    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"]
Nov 03 08:19:25.530367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.530343    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0fc79b-dockercfg-926p2\""
Nov 03 08:19:25.530497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.530405    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0fc79b\""
Nov 03 08:19:25.564790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.564757    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"]
Nov 03 08:19:25.569717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.569695    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:25.576984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.576955    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"]
Nov 03 08:19:25.608475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.608422    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:25.657307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.653821    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ccc85-c392-46c4-ab2f-9e0217071869-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:25.657307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.654077    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jh9\" (UniqueName: \"kubernetes.io/projected/df98ceaa-c373-44b9-86b6-e6141ba81776-kube-api-access-c9jh9\") pod \"payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:25.657307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.654113    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5t97\" (UniqueName: \"kubernetes.io/projected/da8ccc85-c392-46c4-ab2f-9e0217071869-kube-api-access-b5t97\") pod \"payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:25.735136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.654162    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ceaa-c373-44b9-86b6-e6141ba81776-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:25.873091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.850959    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ccc85-c392-46c4-ab2f-9e0217071869-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:25.975084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.974544    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jh9\" (UniqueName: \"kubernetes.io/projected/df98ceaa-c373-44b9-86b6-e6141ba81776-kube-api-access-c9jh9\") pod \"payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:25.975084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.974588    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5t97\" (UniqueName: \"kubernetes.io/projected/da8ccc85-c392-46c4-ab2f-9e0217071869-kube-api-access-b5t97\") pod \"payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:25.975084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.974646    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ceaa-c373-44b9-86b6-e6141ba81776-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:25.990131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.975858    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ceaa-c373-44b9-86b6-e6141ba81776-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:25.990131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:25.988996    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ccc85-c392-46c4-ab2f-9e0217071869-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:26.057088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.044460    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5t97\" (UniqueName: \"kubernetes.io/projected/da8ccc85-c392-46c4-ab2f-9e0217071869-kube-api-access-b5t97\") pod \"payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:26.057088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.045856    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jh9\" (UniqueName: \"kubernetes.io/projected/df98ceaa-c373-44b9-86b6-e6141ba81776-kube-api-access-c9jh9\") pod \"payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:26.071261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.071235    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:26.146926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.146898    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:26.199865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.199832    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:26.458921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.458893    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"]
Nov 03 08:19:26.459305 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:19:26.459284    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf98ceaa_c373_44b9_86b6_e6141ba81776.slice/crio-5a1dbe60ecc12858daf7f3642c4a065cfd4ef8485afeaa9db75af8d41fade35b WatchSource:0}: Error finding container 5a1dbe60ecc12858daf7f3642c4a065cfd4ef8485afeaa9db75af8d41fade35b: Status 404 returned error can't find the container with id 5a1dbe60ecc12858daf7f3642c4a065cfd4ef8485afeaa9db75af8d41fade35b
Nov 03 08:19:26.513755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.513730    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"]
Nov 03 08:19:26.517725 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:19:26.517699    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8ccc85_c392_46c4_ab2f_9e0217071869.slice/crio-1c443644b06313ca2c8712ce2715ea81d4ed9c183c37df340441374f1a3352e8 WatchSource:0}: Error finding container 1c443644b06313ca2c8712ce2715ea81d4ed9c183c37df340441374f1a3352e8: Status 404 returned error can't find the container with id 1c443644b06313ca2c8712ce2715ea81d4ed9c183c37df340441374f1a3352e8
Nov 03 08:19:26.910907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:26.910833    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:27.059444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.059406    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:27.220847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.220808    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" event={"ID":"df98ceaa-c373-44b9-86b6-e6141ba81776","Type":"ContainerStarted","Data":"561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286"}
Nov 03 08:19:27.221044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.220865    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" event={"ID":"df98ceaa-c373-44b9-86b6-e6141ba81776","Type":"ContainerStarted","Data":"5a1dbe60ecc12858daf7f3642c4a065cfd4ef8485afeaa9db75af8d41fade35b"}
Nov 03 08:19:27.221044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.220892    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:27.222619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.222594    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" event={"ID":"da8ccc85-c392-46c4-ab2f-9e0217071869","Type":"ContainerStarted","Data":"ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7"}
Nov 03 08:19:27.222795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.222633    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" event={"ID":"da8ccc85-c392-46c4-ab2f-9e0217071869","Type":"ContainerStarted","Data":"1c443644b06313ca2c8712ce2715ea81d4ed9c183c37df340441374f1a3352e8"}
Nov 03 08:19:27.222795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.222653    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:27.238607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.238562    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" podStartSLOduration=2.238550502 podStartE2EDuration="2.238550502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:19:27.235522289 +0000 UTC m=+1516.738357801" watchObservedRunningTime="2026-11-03 08:19:27.238550502 +0000 UTC m=+1516.741386029"
Nov 03 08:19:27.250750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.250707    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" podStartSLOduration=2.250694219 podStartE2EDuration="2.250694219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:19:27.249039554 +0000 UTC m=+1516.751875069" watchObservedRunningTime="2026-11-03 08:19:27.250694219 +0000 UTC m=+1516.753529734"
Nov 03 08:19:27.960897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:27.960865    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:28.185606 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:28.185578    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:19:28.378250 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:19:28.378170    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:19:28.840697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:28.840632    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:29.548695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:29.548650    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:30.281054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:30.281012    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:31.146089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:31.139237    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:31.848200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:31.848167    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:32.649908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:32.649879    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:33.454668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:33.454634    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:34.234773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:34.234729    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:35.038289 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:35.038244    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:35.737268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:35.737238    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:36.442924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:36.442891    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:37.246577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:37.246541    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:38.152941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:38.152905    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:38.238218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:38.238167    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:19:38.238420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:38.238239    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:19:38.744761 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:38.744732    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:39.537528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:39.537491    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:40.250227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:40.250193    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:40.937022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:40.936988    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:41.736125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:41.736090    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:42.708055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:42.708019    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:42.832825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:42.832779    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:43.560169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:43.560139    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:44.348054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:44.348023    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:45.417531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:45.417482    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:45.529183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:45.529148    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:46.728183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:46.728148    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:47.807909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:47.807862    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:48.707000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:48.706959    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:49.008912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:49.008829    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:49.477408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:49.477375    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:49.647105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:49.647040    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:49.933775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:49.933742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:50.123085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:50.120139    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:50.733973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:50.733931    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:51.484644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:51.484505    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:52.252036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:52.252000    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:53.069105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:53.069041    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:53.780437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:53.780410    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:54.545980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:54.545942    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:55.452504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:55.452473    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:56.198086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:56.184098    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:57.010395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:57.010355    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:57.759550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:57.759515    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:58.379804 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:19:58.379770    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:19:58.531656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:58.531626    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:59.356683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:59.356644    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:19:59.974940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:19:59.974905    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:00.752164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:00.752136    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:01.573695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:01.573665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:02.370850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:02.370820    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:03.236602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:03.236569    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:04.047520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:04.047458    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:04.829277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:04.829247    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:05.551125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:05.551056    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:06.245688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:06.245650    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:07.002197 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:07.002159    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:07.791740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:07.791670    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:08.578830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:08.578785    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:09.334910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:09.334878    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:10.045137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:10.045109    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:10.737564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:10.737536    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:11.589086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:11.588652    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:12.397034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:12.397001    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:13.095895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:13.095853    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:13.709155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:13.709106    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:14.009078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:14.008959    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:14.743686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:14.743649    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:15.456384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:15.456339    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:16.165127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:16.165091    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:16.854769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:16.854735    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:17.561520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:17.561483    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:18.490648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:18.490614    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:19.201501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:19.189895    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:19.942083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:19.942021    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:20.749704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:20.749664    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:21.441263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:21.441233    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:22.247365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:22.247331    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:22.971372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:22.971336    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:23.085207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:23.085169    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"]
Nov 03 08:20:23.094488 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:23.094441    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bae2c8-5l7pd"]
Nov 03 08:20:23.743687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:23.743654    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:24.449612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:24.449578    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:24.621463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:24.621423    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4034aaa5-c9b9-4b51-b163-981403892ed6" path="/var/lib/kubelet/pods/4034aaa5-c9b9-4b51-b163-981403892ed6/volumes"
Nov 03 08:20:25.347819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:25.347787    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:26.132724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:26.132688    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:26.845494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:26.845455    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:27.540385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:27.540353    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:28.378353 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:20:28.378319    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:20:28.440976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:28.440941    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:29.154590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:29.154557    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:29.942838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:29.942803    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:30.371746 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:30.371666    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:30.505600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:30.505557    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:30.752971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:30.752936    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:31.817349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:31.816985    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:32.953340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:32.953307    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:34.540320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:34.540274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:35.495212 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:35.495178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:36.240303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:36.240264    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:37.169329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:37.169283    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:38.042153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:38.042119    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:38.770030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:38.770001    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:39.443857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:39.443802    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:40.242248 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:40.242209    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:41.074345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:41.074312    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:41.844055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:41.844019    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:42.388649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:42.388619    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"]
Nov 03 08:20:42.412996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:42.412966    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"]
Nov 03 08:20:42.506142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:42.506110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:42.630841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:42.630812    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:43.246646 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:43.246600    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:44.091845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:44.091808    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:44.403462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:44.403380    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:44.721523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:44.721453    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:44.872270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:44.872214    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:45.219627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:45.219596    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:45.833780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:45.833740    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:46.945092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:46.944584    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:47.389790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.389716    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf" podUID="395b8892-0a9e-4e4a-9885-44f6c18d5c89" containerName="payload-processing" containerID="cri-o://86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2" gracePeriod=25
Nov 03 08:20:47.413616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.413554    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w" podUID="9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" containerName="payload-pre-processing" containerID="cri-o://35771ccde7d9220d4ccdeb0df04c3717744eb7e52a39f317ebdf1e93376b7cfb" gracePeriod=25
Nov 03 08:20:47.727215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.727189    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:20:47.753416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.753382    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:47.754517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.754495    6430 generic.go:356] "Generic (PLEG): container finished" podID="9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" containerID="35771ccde7d9220d4ccdeb0df04c3717744eb7e52a39f317ebdf1e93376b7cfb" exitCode=0
Nov 03 08:20:47.754637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.754567    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w" event={"ID":"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8","Type":"ContainerDied","Data":"35771ccde7d9220d4ccdeb0df04c3717744eb7e52a39f317ebdf1e93376b7cfb"}
Nov 03 08:20:47.754637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.754620    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w" event={"ID":"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8","Type":"ContainerDied","Data":"243a06af110c331bc4b00532f74bac25af1a2f91502367dad5cfa51c76859f3f"}
Nov 03 08:20:47.754762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.754638    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="243a06af110c331bc4b00532f74bac25af1a2f91502367dad5cfa51c76859f3f"
Nov 03 08:20:47.756238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.756216    6430 generic.go:356] "Generic (PLEG): container finished" podID="395b8892-0a9e-4e4a-9885-44f6c18d5c89" containerID="86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2" exitCode=0
Nov 03 08:20:47.756396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.756268    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"
Nov 03 08:20:47.756396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.756303    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf" event={"ID":"395b8892-0a9e-4e4a-9885-44f6c18d5c89","Type":"ContainerDied","Data":"86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2"}
Nov 03 08:20:47.756396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.756361    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf" event={"ID":"395b8892-0a9e-4e4a-9885-44f6c18d5c89","Type":"ContainerDied","Data":"f2a613e77dd25d291c9c507f6fed56041f705c70b2ed4bf350fb9e448d49db69"}
Nov 03 08:20:47.756396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.756389    6430 scope.go:122] "RemoveContainer" containerID="86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2"
Nov 03 08:20:47.783089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.774581    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:20:47.881890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.881855    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-plugins-config-volume\") pod \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") "
Nov 03 08:20:47.882146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.882030    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395b8892-0a9e-4e4a-9885-44f6c18d5c89-kube-api-access-ht9xw\" (UniqueName: \"kubernetes.io/projected/395b8892-0a9e-4e4a-9885-44f6c18d5c89-kube-api-access-ht9xw\") pod \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") "
Nov 03 08:20:47.882146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.882131    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/395b8892-0a9e-4e4a-9885-44f6c18d5c89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395b8892-0a9e-4e4a-9885-44f6c18d5c89-plugins-config-volume\") pod \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\" (UID: \"395b8892-0a9e-4e4a-9885-44f6c18d5c89\") "
Nov 03 08:20:47.882287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.882204    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-kube-api-access-9rsjs\" (UniqueName: \"kubernetes.io/projected/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-kube-api-access-9rsjs\") pod \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\" (UID: \"9fe1ae87-a384-429c-bb08-cc0f2a37fcb8\") "
Nov 03 08:20:47.882287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.882263    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-plugins-config-volume" pod "9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" (UID: "9fe1ae87-a384-429c-bb08-cc0f2a37fcb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:20:47.883152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.882664    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395b8892-0a9e-4e4a-9885-44f6c18d5c89-plugins-config-volume" pod "395b8892-0a9e-4e4a-9885-44f6c18d5c89" (UID: "395b8892-0a9e-4e4a-9885-44f6c18d5c89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:20:47.883386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.883362    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/395b8892-0a9e-4e4a-9885-44f6c18d5c89-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:47.883461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.883394    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:47.885360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.885334    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-kube-api-access-9rsjs" pod "9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" (UID: "9fe1ae87-a384-429c-bb08-cc0f2a37fcb8"). InnerVolumeSpecName "kube-api-access-9rsjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:20:47.885479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.885430    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395b8892-0a9e-4e4a-9885-44f6c18d5c89-kube-api-access-ht9xw" pod "395b8892-0a9e-4e4a-9885-44f6c18d5c89" (UID: "395b8892-0a9e-4e4a-9885-44f6c18d5c89"). InnerVolumeSpecName "kube-api-access-ht9xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:20:47.896267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.896192    6430 scope.go:122] "RemoveContainer" containerID="86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2"
Nov 03 08:20:47.896747 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:20:47.896722    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2\": container with ID starting with 86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2 not found: ID does not exist" containerID="86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2"
Nov 03 08:20:47.896851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.896754    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2"} err="failed to get container status \"86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2\": rpc error: code = NotFound desc = could not find container \"86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2\": container with ID starting with 86a9f590dcfc6321abffcc73fc24c5f1a1b921b8667bf6817d94545cd73cabe2 not found: ID does not exist"
Nov 03 08:20:47.984886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.984821    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9xw\" (UniqueName: \"kubernetes.io/projected/395b8892-0a9e-4e4a-9885-44f6c18d5c89-kube-api-access-ht9xw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:47.984886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:47.984848    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rsjs\" (UniqueName: \"kubernetes.io/projected/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8-kube-api-access-9rsjs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:48.080203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:48.080174    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"]
Nov 03 08:20:48.084503 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:48.084481    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-27c674-766997d76c-nnxgf"]
Nov 03 08:20:48.441760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:48.441730    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:48.620633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:48.620603    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="395b8892-0a9e-4e4a-9885-44f6c18d5c89" path="/var/lib/kubelet/pods/395b8892-0a9e-4e4a-9885-44f6c18d5c89/volumes"
Nov 03 08:20:48.764917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:48.764889    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"
Nov 03 08:20:48.780890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:48.780859    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"]
Nov 03 08:20:48.786775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:48.786749    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-27c674-7f85f4b667-sx47w"]
Nov 03 08:20:49.281445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:49.281412    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:50.059909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:50.059875    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:50.621731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:50.621696    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" path="/var/lib/kubelet/pods/9fe1ae87-a384-429c-bb08-cc0f2a37fcb8/volumes"
Nov 03 08:20:50.869944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:50.869910    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:51.699449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:51.699414    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:52.048832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:52.048750    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:52.183774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:52.183705    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:52.386824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:52.386735    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:53.015122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:53.015080    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:54.347812 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:54.347779    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:55.320429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:55.320389    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:56.065195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:56.065161    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:56.203101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:56.203046    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:56.563814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:56.563781    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:56.928529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:56.928439    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:57.154091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:57.137641    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:57.897865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:57.897828    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:58.379138 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:20:58.379099    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:20:58.672376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:58.672272    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:20:59.555897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:20:59.555869    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:00.360485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:00.360431    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:01.349872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:01.349830    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:01.780573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:01.780538    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:01.980993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:01.980963    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:02.549628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:02.549595    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:02.985209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:02.985182    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:03.011175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.011140    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"]
Nov 03 08:21:03.011964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.011948    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" containerName="payload-pre-processing"
Nov 03 08:21:03.012021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.011966    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" containerName="payload-pre-processing"
Nov 03 08:21:03.012021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.011996    6430 cpu_manager.go:414] "containerMap: removing container" podUID="395b8892-0a9e-4e4a-9885-44f6c18d5c89" containerName="payload-processing"
Nov 03 08:21:03.012021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.012001    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="395b8892-0a9e-4e4a-9885-44f6c18d5c89" containerName="payload-processing"
Nov 03 08:21:03.012142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.012131    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe1ae87-a384-429c-bb08-cc0f2a37fcb8" containerName="payload-pre-processing"
Nov 03 08:21:03.012176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.012146    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="395b8892-0a9e-4e4a-9885-44f6c18d5c89" containerName="payload-processing"
Nov 03 08:21:03.016340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.016321    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.029353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.029328    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-109070\""
Nov 03 08:21:03.058082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.058026    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"]
Nov 03 08:21:03.062599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.062572    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"]
Nov 03 08:21:03.068996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.068967    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:03.070668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.070644    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"]
Nov 03 08:21:03.071974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.071951    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-109070-dockercfg-sh6nd\""
Nov 03 08:21:03.121573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.121527    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:03.127667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.127633    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dcl\" (UniqueName: \"kubernetes.io/projected/2d61781c-fa71-42ef-8ada-4350c32675bd-kube-api-access-52dcl\") pod \"payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.128337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.128311    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d61781c-fa71-42ef-8ada-4350c32675bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.246805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.236390    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d61781c-fa71-42ef-8ada-4350c32675bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.252547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.252509    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dcl\" (UniqueName: \"kubernetes.io/projected/2d61781c-fa71-42ef-8ada-4350c32675bd-kube-api-access-52dcl\") pod \"payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.271996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.271957    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6jr\" (UniqueName: \"kubernetes.io/projected/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-kube-api-access-9b6jr\") pod \"payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:03.273386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.273357    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:03.273721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.273085    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d61781c-fa71-42ef-8ada-4350c32675bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.355800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.355763    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dcl\" (UniqueName: \"kubernetes.io/projected/2d61781c-fa71-42ef-8ada-4350c32675bd-kube-api-access-52dcl\") pod \"payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.375736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.375690    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6jr\" (UniqueName: \"kubernetes.io/projected/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-kube-api-access-9b6jr\") pod \"payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:03.376137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.376115    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:03.377577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.377555    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:03.413311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.413277    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6jr\" (UniqueName: \"kubernetes.io/projected/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-kube-api-access-9b6jr\") pod \"payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:03.639903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.639824    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:03.687043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.687010    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:04.004120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:03.997518    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"]
Nov 03 08:21:04.047819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:04.047794    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:04.057937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:04.057918    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"]
Nov 03 08:21:04.059559 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:21:04.059525    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b0ee03_41b9_4acf_9e37_8080afa27f1c.slice/crio-b8d5c63be5d77a1eb6f9cd11cf77e1c1d68f32056c81785abdaeb24105438336 WatchSource:0}: Error finding container b8d5c63be5d77a1eb6f9cd11cf77e1c1d68f32056c81785abdaeb24105438336: Status 404 returned error can't find the container with id b8d5c63be5d77a1eb6f9cd11cf77e1c1d68f32056c81785abdaeb24105438336
Nov 03 08:21:04.068124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:04.068096    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" event={"ID":"2d61781c-fa71-42ef-8ada-4350c32675bd","Type":"ContainerStarted","Data":"f2c65979138eadd947a7cf577825e2774b3e146f54ddfd86a12bcabd9dfcca27"}
Nov 03 08:21:04.918955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:04.918903    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:05.078492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.078455    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" event={"ID":"2d61781c-fa71-42ef-8ada-4350c32675bd","Type":"ContainerStarted","Data":"6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da"}
Nov 03 08:21:05.078679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.078517    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:05.080230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.080204    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" event={"ID":"a7b0ee03-41b9-4acf-9e37-8080afa27f1c","Type":"ContainerStarted","Data":"26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e"}
Nov 03 08:21:05.080355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.080252    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" event={"ID":"a7b0ee03-41b9-4acf-9e37-8080afa27f1c","Type":"ContainerStarted","Data":"b8d5c63be5d77a1eb6f9cd11cf77e1c1d68f32056c81785abdaeb24105438336"}
Nov 03 08:21:05.080355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.080280    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:05.095574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.095518    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" podStartSLOduration=3.095502031 podStartE2EDuration="3.095502031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:05.093628428 +0000 UTC m=+1614.596463945" watchObservedRunningTime="2026-11-03 08:21:05.095502031 +0000 UTC m=+1614.598337547"
Nov 03 08:21:05.112848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.112807    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" podStartSLOduration=2.1127959179999998 podStartE2EDuration="2.112795918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:05.109615703 +0000 UTC m=+1614.612451219" watchObservedRunningTime="2026-11-03 08:21:05.112795918 +0000 UTC m=+1614.615631431"
Nov 03 08:21:05.866521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:05.866484    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:06.661027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:06.660998    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:07.440675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:07.440643    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:08.244112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:08.244053    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:08.487215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:08.487158    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:08.606370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:08.606297    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:08.971496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:08.971467    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:09.739899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:09.739866    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:10.502893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:10.502849    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:11.170349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:11.170313    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:11.874406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:11.874378    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:12.623525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:12.623494    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:13.571098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:13.563608    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:13.823696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:13.823606    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:14.290628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:14.290579    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:15.101095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:15.100730    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:15.881452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:15.881416    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:16.124858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:16.124833    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:21:16.125280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:16.124897    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:21:16.664611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:16.664574    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:17.116802 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.116767    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"]
Nov 03 08:21:17.117038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.117014    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy" containerID="cri-o://13c1f47c243249528f3d051efae7a9387397bc51c145d14b2578bcfa7105176d" gracePeriod=30
Nov 03 08:21:17.429661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.429567    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"]
Nov 03 08:21:17.430210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.429847    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy" containerID="cri-o://0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580" gracePeriod=30
Nov 03 08:21:17.489145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.489110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:17.678448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.678404    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"]
Nov 03 08:21:17.678957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.678927    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy" containerID="cri-o://371c029c2651a6c9d9dff6b049744c4e36ef7885f0cb291241a3502a31a6c1c4" gracePeriod=30
Nov 03 08:21:17.771886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.771846    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"]
Nov 03 08:21:17.772654 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.772367    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy" containerID="cri-o://840f7db386520802a8db18e041fb4644b652caa4673b05380541632565129671" gracePeriod=30
Nov 03 08:21:17.775759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.775729    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"]
Nov 03 08:21:17.776046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:17.776023    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy" containerID="cri-o://2d77e5ea66c12dfe20b916e99a8974b59170a17131414f5a1d0d12f582519758" gracePeriod=30
Nov 03 08:21:18.295780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:18.295717    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:19.070819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:19.070779    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:19.847838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:19.847780    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:20.599202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:20.599162    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:21.258973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:21.258938    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:22.135003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.134973    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:22.300998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.300785    6430 generic.go:356] "Generic (PLEG): container finished" podID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerID="13c1f47c243249528f3d051efae7a9387397bc51c145d14b2578bcfa7105176d" exitCode=0
Nov 03 08:21:22.300998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.300940    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" event={"ID":"80e103cd-753b-4ca4-be50-7a17dfe55440","Type":"ContainerDied","Data":"13c1f47c243249528f3d051efae7a9387397bc51c145d14b2578bcfa7105176d"}
Nov 03 08:21:22.558670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.558638    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:21:22.590953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.590909    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"]
Nov 03 08:21:22.592234 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.592195    6430 cpu_manager.go:414] "containerMap: removing container" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy"
Nov 03 08:21:22.592234 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.592221    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy"
Nov 03 08:21:22.592507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.592491    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" containerName="istio-proxy"
Nov 03 08:21:22.596383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.596363    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.600590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.600573    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b8a152-openshift-default-dockercfg-pksss\""
Nov 03 08:21:22.642771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.642738    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"]
Nov 03 08:21:22.709658 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.709573    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e103cd-753b-4ca4-be50-7a17dfe55440-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e103cd-753b-4ca4-be50-7a17dfe55440-istiod-ca-cert\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.714410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.710264    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e103cd-753b-4ca4-be50-7a17dfe55440-istiod-ca-cert" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:22.715126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.715106    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-envoy\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.715329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.715311    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-certs\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.715820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.715803    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-kube-api-access-qqlpb\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-kube-api-access-qqlpb\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.717236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.717214    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-token\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-token\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.720232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.715674    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-certs" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.720586 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.720565    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-credential-socket\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.720809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.720793    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-data\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.721000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.720983    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-socket\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.721178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.721162    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-podinfo\") pod \"80e103cd-753b-4ca4-be50-7a17dfe55440\" (UID: \"80e103cd-753b-4ca4-be50-7a17dfe55440\") "
Nov 03 08:21:22.722455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.722409    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.723257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.722678    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-credential-socket" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.723257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.722847    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-socket" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.723257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.723244    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-data" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.723853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.723766    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.724037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.724018    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.724310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.724290    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.724453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.724435    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.724631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.724614    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.724830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.724813    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.725148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.725126    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.725341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.725322    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566k6\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-kube-api-access-566k6\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.725677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.725655    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.725807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.725794    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.725908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.725895    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.726025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.726004    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e103cd-753b-4ca4-be50-7a17dfe55440-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.726198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.726176    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.726642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.726616    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-kube-api-access-qqlpb" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "kube-api-access-qqlpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:22.735503 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.735386    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-envoy" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.735503 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.735426    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-token" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:22.736769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.736750    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-podinfo" pod "80e103cd-753b-4ca4-be50-7a17dfe55440" (UID: "80e103cd-753b-4ca4-be50-7a17dfe55440"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:22.830148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830107    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566k6\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-kube-api-access-566k6\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830257    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830360    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830458    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830523    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830555    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830622    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830681    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.830847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830837    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.832921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830959    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.832921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830980    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.832921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.830999    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqlpb\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-kube-api-access-qqlpb\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.832921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.831016    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e103cd-753b-4ca4-be50-7a17dfe55440-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.832921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.831456    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.832921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.832076    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.832921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.832318    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.833317 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.833030    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.833968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.833937    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.836970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.836931    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.840256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.840224    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.844208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.844185    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566k6\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-kube-api-access-566k6\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.845039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.845020    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.909461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.909423    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:21:22.938662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.938385    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:22.984188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.984160    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"]
Nov 03 08:21:22.985448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.985419    6430 cpu_manager.go:414] "containerMap: removing container" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy"
Nov 03 08:21:22.985448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.985446    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy"
Nov 03 08:21:22.985716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.985701    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerName="istio-proxy"
Nov 03 08:21:22.998104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.998076    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"]
Nov 03 08:21:22.998507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:22.998399    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.010156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.008868    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-109070-serving-cert\""
Nov 03 08:21:23.016958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.016871    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.039744    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-certs\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.039921    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-credential-socket\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.040123    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-data\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.040457    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-credential-socket" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.040802    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-data" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.040990    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-kube-api-access-n49qp\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-kube-api-access-n49qp\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.041398    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-certs" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.041440    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-podinfo\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.041536    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68451d0b-c751-4b74-a72c-2dcd31d07254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68451d0b-c751-4b74-a72c-2dcd31d07254-istiod-ca-cert\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.041645    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-token\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-token\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.041707    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-envoy\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.042106    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68451d0b-c751-4b74-a72c-2dcd31d07254-istiod-ca-cert" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.042422    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-socket\") pod \"68451d0b-c751-4b74-a72c-2dcd31d07254\" (UID: \"68451d0b-c751-4b74-a72c-2dcd31d07254\") "
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.043462    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.043483    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.043497    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.043514    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68451d0b-c751-4b74-a72c-2dcd31d07254-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.044084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.043799    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-socket" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.053994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.047052    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-envoy" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.074662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.074618    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-podinfo" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:23.078015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.077979    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-token" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.091001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.090940    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-kube-api-access-n49qp" pod "68451d0b-c751-4b74-a72c-2dcd31d07254" (UID: "68451d0b-c751-4b74-a72c-2dcd31d07254"). InnerVolumeSpecName "kube-api-access-n49qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.139119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.138953    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.145319    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.145632    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51912a37-547f-4db7-b5e9-a56b62cfa3bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.145678    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-metrics-certs\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.145783    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hbc\" (UniqueName: \"kubernetes.io/projected/51912a37-547f-4db7-b5e9-a56b62cfa3bf-kube-api-access-t6hbc\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.146159    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n49qp\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-kube-api-access-n49qp\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.146518    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.146540    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.146554    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.146862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.146568    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68451d0b-c751-4b74-a72c-2dcd31d07254-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.306184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.298359    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51912a37-547f-4db7-b5e9-a56b62cfa3bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.306184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.298426    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-metrics-certs\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.306184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.298509    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hbc\" (UniqueName: \"kubernetes.io/projected/51912a37-547f-4db7-b5e9-a56b62cfa3bf-kube-api-access-t6hbc\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.306184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.298599    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.313401    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51912a37-547f-4db7-b5e9-a56b62cfa3bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.322587    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-metrics-certs\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.323755    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.326033    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hbc\" (UniqueName: \"kubernetes.io/projected/51912a37-547f-4db7-b5e9-a56b62cfa3bf-kube-api-access-t6hbc\") pod \"maas-api-e2e-inf-b-109070-5bf789b96-xrbj5\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.345322    6430 generic.go:356] "Generic (PLEG): container finished" podID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerID="371c029c2651a6c9d9dff6b049744c4e36ef7885f0cb291241a3502a31a6c1c4" exitCode=0
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.345411    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" event={"ID":"05e482cc-190c-4a3f-aa6f-59ff75972735","Type":"ContainerDied","Data":"371c029c2651a6c9d9dff6b049744c4e36ef7885f0cb291241a3502a31a6c1c4"}
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.345463    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp" event={"ID":"05e482cc-190c-4a3f-aa6f-59ff75972735","Type":"ContainerDied","Data":"455b5aa215037dd3466e80c42b4e6abe46e0e47c8afbee0668cd5fb05bd2bc07"}
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.345488    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="455b5aa215037dd3466e80c42b4e6abe46e0e47c8afbee0668cd5fb05bd2bc07"
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.347834    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t" event={"ID":"80e103cd-753b-4ca4-be50-7a17dfe55440","Type":"ContainerDied","Data":"daf39b5ba9ff30f7e04610aff5c8358d829cc4b67e45c91bb90dc23ee755f839"}
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.347895    6430 scope.go:122] "RemoveContainer" containerID="13c1f47c243249528f3d051efae7a9387397bc51c145d14b2578bcfa7105176d"
Nov 03 08:21:23.377433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.348133    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"
Nov 03 08:21:23.434123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.432993    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:23.450110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.450037    6430 generic.go:356] "Generic (PLEG): container finished" podID="4423db23-dc99-4360-b61e-87b6b323abe5" containerID="2d77e5ea66c12dfe20b916e99a8974b59170a17131414f5a1d0d12f582519758" exitCode=0
Nov 03 08:21:23.450696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.450662    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" event={"ID":"4423db23-dc99-4360-b61e-87b6b323abe5","Type":"ContainerDied","Data":"2d77e5ea66c12dfe20b916e99a8974b59170a17131414f5a1d0d12f582519758"}
Nov 03 08:21:23.453676 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.453646    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"]
Nov 03 08:21:23.454035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.453983    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy" containerID="cri-o://e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894" gracePeriod=30
Nov 03 08:21:23.457795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.457278    6430 generic.go:356] "Generic (PLEG): container finished" podID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerID="840f7db386520802a8db18e041fb4644b652caa4673b05380541632565129671" exitCode=0
Nov 03 08:21:23.457795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.457414    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" event={"ID":"5138aa05-5757-40de-b2d9-82bce662ec9f","Type":"ContainerDied","Data":"840f7db386520802a8db18e041fb4644b652caa4673b05380541632565129671"}
Nov 03 08:21:23.461568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.461536    6430 generic.go:356] "Generic (PLEG): container finished" podID="68451d0b-c751-4b74-a72c-2dcd31d07254" containerID="0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580" exitCode=0
Nov 03 08:21:23.461711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.461691    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" event={"ID":"68451d0b-c751-4b74-a72c-2dcd31d07254","Type":"ContainerDied","Data":"0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580"}
Nov 03 08:21:23.461794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.461750    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm" event={"ID":"68451d0b-c751-4b74-a72c-2dcd31d07254","Type":"ContainerDied","Data":"a1d7a88523d0aed89a0be98689c755fb00fbfc0bcecd82492fbc67d4e8ac96b0"}
Nov 03 08:21:23.461918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.461876    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"
Nov 03 08:21:23.473789 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.473758    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:21:23.476568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.476042    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:21:23.485942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.485918    6430 scope.go:122] "RemoveContainer" containerID="0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580"
Nov 03 08:21:23.509253 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.509227    6430 scope.go:122] "RemoveContainer" containerID="0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580"
Nov 03 08:21:23.511332 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:21:23.509965    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580\": container with ID starting with 0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580 not found: ID does not exist" containerID="0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580"
Nov 03 08:21:23.511332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.510005    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580"} err="failed to get container status \"0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580\": rpc error: code = NotFound desc = could not find container \"0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580\": container with ID starting with 0be4ede3195621a30e0a9ebf0bd43ff826f5b0903ccb6ac26211d1190278c580 not found: ID does not exist"
Nov 03 08:21:23.513956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.513937    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:21:23.557724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.557639    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4423db23-dc99-4360-b61e-87b6b323abe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4423db23-dc99-4360-b61e-87b6b323abe5-istiod-ca-cert\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.557724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.557703    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-podinfo\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.557947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.557755    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-credential-socket\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.557947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.557784    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-data\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.557947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.557871    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-credential-socket\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.557947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.557900    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4423db23-dc99-4360-b61e-87b6b323abe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4423db23-dc99-4360-b61e-87b6b323abe5-istio-podinfo\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.557947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.557936    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-socket\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.558233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558009    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-kube-api-access-p6ptd\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-kube-api-access-p6ptd\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.558233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558082    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-envoy\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558111    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-istio-token\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-istio-token\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.558233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558150    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e482cc-190c-4a3f-aa6f-59ff75972735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e482cc-190c-4a3f-aa6f-59ff75972735-istiod-ca-cert\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558192    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-kube-api-access-bxclw\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-kube-api-access-bxclw\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558262    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-socket\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.558515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558303    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-token\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-token\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.558515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558390    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-kube-api-access-j9z59\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-kube-api-access-j9z59\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.558515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558441    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-podinfo\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558490    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-credential-socket\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.558754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558558    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-data\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.558754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558625    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-certs\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558743    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-socket\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558772    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-token\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-token\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558825    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-data\") pod \"05e482cc-190c-4a3f-aa6f-59ff75972735\" (UID: \"05e482cc-190c-4a3f-aa6f-59ff75972735\") "
Nov 03 08:21:23.558912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558882    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-envoy\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.559132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558920    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-certs\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.559132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.558952    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-certs\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.559132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.559088    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-envoy\") pod \"4423db23-dc99-4360-b61e-87b6b323abe5\" (UID: \"4423db23-dc99-4360-b61e-87b6b323abe5\") "
Nov 03 08:21:23.559132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.559127    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5138aa05-5757-40de-b2d9-82bce662ec9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138aa05-5757-40de-b2d9-82bce662ec9f-istiod-ca-cert\") pod \"5138aa05-5757-40de-b2d9-82bce662ec9f\" (UID: \"5138aa05-5757-40de-b2d9-82bce662ec9f\") "
Nov 03 08:21:23.566430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.564302    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-socket" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.566430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.564330    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-credential-socket" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.566430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.564491    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-credential-socket" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.566430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.564633    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-data" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.566430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.564686    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4423db23-dc99-4360-b61e-87b6b323abe5-istiod-ca-cert" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:23.566430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.564768    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-certs" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.566430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.565218    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-data" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.566775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.566463    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-socket" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.570400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.570373    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-socket" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.583213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.572643    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-certs" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.583213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.572794    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-certs" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.583213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.579721    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e482cc-190c-4a3f-aa6f-59ff75972735-istiod-ca-cert" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:23.583213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.582360    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-credential-socket" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.583213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.582580    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5138aa05-5757-40de-b2d9-82bce662ec9f-istiod-ca-cert" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:23.587334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.587298    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-data" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.590408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.590389    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"]
Nov 03 08:21:23.597374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.597352    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy"
Nov 03 08:21:23.599304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.599283    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy"
Nov 03 08:21:23.599470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.599456    6430 cpu_manager.go:414] "containerMap: removing container" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy"
Nov 03 08:21:23.599581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.599567    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy"
Nov 03 08:21:23.599737 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.599724    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy"
Nov 03 08:21:23.599831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.599819    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy"
Nov 03 08:21:23.600200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.600183    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" containerName="istio-proxy"
Nov 03 08:21:23.600385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.600371    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" containerName="istio-proxy"
Nov 03 08:21:23.600494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.600483    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" containerName="istio-proxy"
Nov 03 08:21:23.614631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.612566    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"]
Nov 03 08:21:23.614631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.612685    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.616693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.616423    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-envoy" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.616693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.616472    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-token" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.649470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.621476    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"]
Nov 03 08:21:23.649470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.643735    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-token" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.651044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.650994    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-envoy" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.651368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.651333    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-podinfo" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:23.651591 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.651559    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4423db23-dc99-4360-b61e-87b6b323abe5-istio-podinfo" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:23.661966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.657144    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-kube-api-access-bxclw" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "kube-api-access-bxclw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.661966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.657784    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-envoy" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:23.661966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.657888    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-istio-token" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.661966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.657940    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-podinfo" pod "05e482cc-190c-4a3f-aa6f-59ff75972735" (UID: "05e482cc-190c-4a3f-aa6f-59ff75972735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:23.665387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.665357    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.675426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.675402    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.676047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.676026    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacf2a1-9660-4dc5-beec-3406a32e24fe-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.678083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.676314    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.678391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.678371    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.679037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.679017    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.679742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.679722    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cgs\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-kube-api-access-r6cgs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.681190 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.681170    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.681360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.681345    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.681638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.681623    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.681740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.681726    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.681833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.681822    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.682382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.682366    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.693095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.693049    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.693273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.693258    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.693373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.693361    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.693908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.693893    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.694037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.694025    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.694168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.694156    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138aa05-5757-40de-b2d9-82bce662ec9f-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.694281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.694269    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4423db23-dc99-4360-b61e-87b6b323abe5-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.694392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.694380    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.694500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.694490    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.695739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.695720    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.695871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.695856    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.695966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.695954    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4423db23-dc99-4360-b61e-87b6b323abe5-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.696077 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.696049    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4423db23-dc99-4360-b61e-87b6b323abe5-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.696243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.696231    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.696347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.696335    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.696423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.692557    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.696657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.692476    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"]
Nov 03 08:21:23.696817 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.696804    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-ccj5t"]
Nov 03 08:21:23.696967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.696955    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"]
Nov 03 08:21:23.698395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.698381    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"]
Nov 03 08:21:23.698558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.698546    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-44lpm"]
Nov 03 08:21:23.698765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.691210    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-kube-api-access-p6ptd" pod "5138aa05-5757-40de-b2d9-82bce662ec9f" (UID: "5138aa05-5757-40de-b2d9-82bce662ec9f"). InnerVolumeSpecName "kube-api-access-p6ptd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.699043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.697721    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-kube-api-access-j9z59" pod "4423db23-dc99-4360-b61e-87b6b323abe5" (UID: "4423db23-dc99-4360-b61e-87b6b323abe5"). InnerVolumeSpecName "kube-api-access-j9z59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:23.699183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.696415    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e482cc-190c-4a3f-aa6f-59ff75972735-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.699374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.699361    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxclw\" (UniqueName: \"kubernetes.io/projected/05e482cc-190c-4a3f-aa6f-59ff75972735-kube-api-access-bxclw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.699573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.699559    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.699761 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.699750    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.700116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.700101    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e482cc-190c-4a3f-aa6f-59ff75972735-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.700256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.700243    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138aa05-5757-40de-b2d9-82bce662ec9f-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.703370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.703329    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7f6f94-serving-cert\""
Nov 03 08:21:23.719489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.717498    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"]
Nov 03 08:21:23.729388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.728298    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.740085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.738929    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"]
Nov 03 08:21:23.788235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.788203    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"]
Nov 03 08:21:23.795861 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:21:23.795837    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce14510d_b9f4_4d27_9f20_38ca6d0cbf8f.slice/crio-d92013637365235c18322eadb4647bf544503fdd4ca7e2b75904c881d6a95525 WatchSource:0}: Error finding container d92013637365235c18322eadb4647bf544503fdd4ca7e2b75904c881d6a95525: Status 404 returned error can't find the container with id d92013637365235c18322eadb4647bf544503fdd4ca7e2b75904c881d6a95525
Nov 03 08:21:23.800343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.800303    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:23.800459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.800394    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:23.800459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.800426    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:23.803547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.803510    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0c8a52-serving-cert\""
Nov 03 08:21:23.804617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804550    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ff3ca9-1574-4328-9ec3-d942c5941e1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.804617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804602    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.804742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804621    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.804742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804657    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zslr\" (UniqueName: \"kubernetes.io/projected/85ff3ca9-1574-4328-9ec3-d942c5941e1a-kube-api-access-7zslr\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.804742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804719    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacf2a1-9660-4dc5-beec-3406a32e24fe-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.804853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804756    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-maas-api-tls\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.804853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804782    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.804853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804809    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-metrics-certs\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.804853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804851    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088e6633-2419-4859-a7c3-cadcffb439d9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.805004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804877    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.805004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804901    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvht\" (UniqueName: \"kubernetes.io/projected/088e6633-2419-4859-a7c3-cadcffb439d9-kube-api-access-tjvht\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.805004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804927    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-metrics-certs\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.805004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804950    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.805004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.804964    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.805208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.805014    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cgs\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-kube-api-access-r6cgs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.805208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.805052    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.805208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.805108    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.805208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.805195    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6ptd\" (UniqueName: \"kubernetes.io/projected/5138aa05-5757-40de-b2d9-82bce662ec9f-kube-api-access-p6ptd\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.805350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.805208    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9z59\" (UniqueName: \"kubernetes.io/projected/4423db23-dc99-4360-b61e-87b6b323abe5-kube-api-access-j9z59\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:23.805867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.805843    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.806208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.806185    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.806447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.806417    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.806916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.806892    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.807479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.807453    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacf2a1-9660-4dc5-beec-3406a32e24fe-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.808175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.808123    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.808815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.808795    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.813756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.813735    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.815822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.815798    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cgs\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-kube-api-access-r6cgs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:23.936655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.936607    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"]
Nov 03 08:21:23.939233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.939190    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvht\" (UniqueName: \"kubernetes.io/projected/088e6633-2419-4859-a7c3-cadcffb439d9-kube-api-access-tjvht\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.939532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.939515    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-metrics-certs\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.939797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.939773    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.940525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.940424    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ff3ca9-1574-4328-9ec3-d942c5941e1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.950085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.941430    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zslr\" (UniqueName: \"kubernetes.io/projected/85ff3ca9-1574-4328-9ec3-d942c5941e1a-kube-api-access-7zslr\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.989137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.950857    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ff3ca9-1574-4328-9ec3-d942c5941e1a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.989137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.951289    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:23.989137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.969982    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-maas-api-tls\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.989137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.970083    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-metrics-certs\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.989137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.970161    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088e6633-2419-4859-a7c3-cadcffb439d9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:23.989137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:23.971081    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088e6633-2419-4859-a7c3-cadcffb439d9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:24.054989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.029525    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:24.054989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.030564    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:21:24.054989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.030955    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-metrics-certs\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:24.054989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.038142    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-maas-api-tls\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:24.056890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.056431    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zslr\" (UniqueName: \"kubernetes.io/projected/85ff3ca9-1574-4328-9ec3-d942c5941e1a-kube-api-access-7zslr\") pod \"maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:24.056890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.056828    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-metrics-certs\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:24.058798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.058612    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvht\" (UniqueName: \"kubernetes.io/projected/088e6633-2419-4859-a7c3-cadcffb439d9-kube-api-access-tjvht\") pod \"maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:24.076539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.066706    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:24.248437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.248410    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:24.442848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.430433    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:24.474621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.471513    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"]
Nov 03 08:21:24.535956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.535914    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:24.536213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.536188    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:24.536362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.536342    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:24.552649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.552621    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx" event={"ID":"4423db23-dc99-4360-b61e-87b6b323abe5","Type":"ContainerDied","Data":"ca768a3880bd3de1b8b72caaa0c0f34848aeeafd2d62fe2a064a49acb51d5664"}
Nov 03 08:21:24.552749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.552706    6430 scope.go:122] "RemoveContainer" containerID="2d77e5ea66c12dfe20b916e99a8974b59170a17131414f5a1d0d12f582519758"
Nov 03 08:21:24.552831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.552819    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"
Nov 03 08:21:24.562977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.562947    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr" event={"ID":"5138aa05-5757-40de-b2d9-82bce662ec9f","Type":"ContainerDied","Data":"b89811cc5b2ba528f6dd32a629d54dcff8a943e31b94a622f986ef6ed1dfedf2"}
Nov 03 08:21:24.563120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.563086    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"
Nov 03 08:21:24.571841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.571817    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" event={"ID":"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f","Type":"ContainerStarted","Data":"e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb"}
Nov 03 08:21:24.571950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.571861    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" event={"ID":"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f","Type":"ContainerStarted","Data":"d92013637365235c18322eadb4647bf544503fdd4ca7e2b75904c881d6a95525"}
Nov 03 08:21:24.579404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.576050    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"
Nov 03 08:21:24.579404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.576081    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5" event={"ID":"51912a37-547f-4db7-b5e9-a56b62cfa3bf","Type":"ContainerStarted","Data":"41d1246e39e7bc4fdc0ed1422c6e95134efb53edd5881d7595402f753955a679"}
Nov 03 08:21:24.588296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.586847    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"]
Nov 03 08:21:24.592537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.591596    6430 scope.go:122] "RemoveContainer" containerID="840f7db386520802a8db18e041fb4644b652caa4673b05380541632565129671"
Nov 03 08:21:24.594422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.594137    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-xpfhx"]
Nov 03 08:21:24.615007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.614959    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podStartSLOduration=154.614941076 podStartE2EDuration="2m34.614941076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:24.609573188 +0000 UTC m=+1634.112408703" watchObservedRunningTime="2026-11-03 08:21:24.614941076 +0000 UTC m=+1634.117776590"
Nov 03 08:21:24.648309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.647892    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4423db23-dc99-4360-b61e-87b6b323abe5" path="/var/lib/kubelet/pods/4423db23-dc99-4360-b61e-87b6b323abe5/volumes"
Nov 03 08:21:24.648744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.648719    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68451d0b-c751-4b74-a72c-2dcd31d07254" path="/var/lib/kubelet/pods/68451d0b-c751-4b74-a72c-2dcd31d07254/volumes"
Nov 03 08:21:24.663704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.660803    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e103cd-753b-4ca4-be50-7a17dfe55440" path="/var/lib/kubelet/pods/80e103cd-753b-4ca4-be50-7a17dfe55440/volumes"
Nov 03 08:21:24.676621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.676596    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"]
Nov 03 08:21:24.684104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.684084    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-rpqmr"]
Nov 03 08:21:24.711210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.711137    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"]
Nov 03 08:21:24.726478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.721141    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-wq7xp"]
Nov 03 08:21:24.726478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.725185    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"]
Nov 03 08:21:24.857085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.856914    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"]
Nov 03 08:21:24.862258 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:21:24.862211    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ff3ca9_1574_4328_9ec3_d942c5941e1a.slice/crio-f6413f9c4bb47efb99c53bfbbbc9588a7c4768c270f252eb3d74eb240ff6fb6e WatchSource:0}: Error finding container f6413f9c4bb47efb99c53bfbbbc9588a7c4768c270f252eb3d74eb240ff6fb6e: Status 404 returned error can't find the container with id f6413f9c4bb47efb99c53bfbbbc9588a7c4768c270f252eb3d74eb240ff6fb6e
Nov 03 08:21:24.939560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:24.939515    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:25.197828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.197794    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:25.588184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.588103    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" event={"ID":"feacf2a1-9660-4dc5-beec-3406a32e24fe","Type":"ContainerStarted","Data":"5db2038d101c0edbba162c3530431c6e87ad83ab1ae34e0ef1127c593a8ef1d2"}
Nov 03 08:21:25.588184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.588153    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" event={"ID":"feacf2a1-9660-4dc5-beec-3406a32e24fe","Type":"ContainerStarted","Data":"44cb11b67735d13248d147e1527937026b80675ca9030a78ca46e4529da452d4"}
Nov 03 08:21:25.590191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.590158    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j" event={"ID":"088e6633-2419-4859-a7c3-cadcffb439d9","Type":"ContainerStarted","Data":"1997108fc3fd43ba6ec65162359a55763b80627f32ef17dd63bd2b60fb92e916"}
Nov 03 08:21:25.590323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.590208    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j" event={"ID":"088e6633-2419-4859-a7c3-cadcffb439d9","Type":"ContainerStarted","Data":"1e76a0a648d9316597d6b15a38c488ca881625043eb6ac68253911c9e971e246"}
Nov 03 08:21:25.591098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.591077    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:25.594474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.594449    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5" event={"ID":"51912a37-547f-4db7-b5e9-a56b62cfa3bf","Type":"ContainerStarted","Data":"8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233"}
Nov 03 08:21:25.595156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.595130    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:25.643152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.641934    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podStartSLOduration=219.64191553 podStartE2EDuration="3m39.64191553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:25.620709322 +0000 UTC m=+1635.123544849" watchObservedRunningTime="2026-11-03 08:21:25.64191553 +0000 UTC m=+1635.144751044"
Nov 03 08:21:25.657088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.646228    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5" event={"ID":"85ff3ca9-1574-4328-9ec3-d942c5941e1a","Type":"ContainerStarted","Data":"f6413f9c4bb47efb99c53bfbbbc9588a7c4768c270f252eb3d74eb240ff6fb6e"}
Nov 03 08:21:25.665093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.664835    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5" podStartSLOduration=3.101225831 podStartE2EDuration="3.66481264s" totalImagesPullingTime="563.586809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:25.64250224 +0000 UTC m=+1635.145337766" watchObservedRunningTime="2026-11-03 08:21:25.66481264 +0000 UTC m=+1635.167648156"
Nov 03 08:21:25.674642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.674580    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j" podStartSLOduration=204.203321846 podStartE2EDuration="3m24.674561503s" totalImagesPullingTime="471.239657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:25.664622 +0000 UTC m=+1635.167457513" watchObservedRunningTime="2026-11-03 08:21:25.674561503 +0000 UTC m=+1635.177397019"
Nov 03 08:21:25.943976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.943934    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 08:21:25.944297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:25.944264    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 08:21:26.053132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.040102    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:26.053132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.043384    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:21:26.053132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.043441    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 08:21:26.666532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.666494    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:26.691451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.691418    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e482cc-190c-4a3f-aa6f-59ff75972735" path="/var/lib/kubelet/pods/05e482cc-190c-4a3f-aa6f-59ff75972735/volumes"
Nov 03 08:21:26.694756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.694728    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5138aa05-5757-40de-b2d9-82bce662ec9f" path="/var/lib/kubelet/pods/5138aa05-5757-40de-b2d9-82bce662ec9f/volumes"
Nov 03 08:21:26.695493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.695471    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:26.695681 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.695662    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5" event={"ID":"85ff3ca9-1574-4328-9ec3-d942c5941e1a","Type":"ContainerStarted","Data":"90cf093368e314bd44db85d744247f64dda82d0b7c11fa86cff6d92ca2cf935f"}
Nov 03 08:21:26.706162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.706035    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5" podStartSLOduration=212.105573891 podStartE2EDuration="3m32.706011413s" totalImagesPullingTime="600.437522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:26.705483963 +0000 UTC m=+1636.208319478" watchObservedRunningTime="2026-11-03 08:21:26.706011413 +0000 UTC m=+1636.208847000"
Nov 03 08:21:26.806086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.803596    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:26.945141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.939508    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 08:21:26.945141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:26.939574    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 08:21:27.037541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:27.037497    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:21:27.037820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:27.037789    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 08:21:27.944384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:27.944348    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 08:21:27.949264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:27.949214    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 08:21:27.957084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:27.947194    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:28.035107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:28.035040    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:21:28.058929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:28.058885    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 08:21:28.258145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:28.254605    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:28.609351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:28.609255    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:28.859430 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:21:28.859331    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:21:28.952143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:28.951606    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 08:21:28.952143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:28.951663    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 08:21:29.056662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.056611    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:21:29.056858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.056680    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 08:21:29.283416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.283389    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:21:29.378441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.378401    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-kube-api-access-85rh6\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-kube-api-access-85rh6\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.378740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.378711    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-credential-socket\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.378877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.378864    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istiod-ca-cert\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.379087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.379050    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-data\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.379252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.379239    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-podinfo\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.379477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.379454    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-envoy\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.379641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.379627    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-socket\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.379843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.379816    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-token\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-token\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.379988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.379974    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-certs\") pod \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\" (UID: \"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5\") "
Nov 03 08:21:29.380471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.380426    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-data" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.381102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.381076    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.381425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.381408    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-certs" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.381609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.381581    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-credential-socket" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.382095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.382053    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istiod-ca-cert" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:29.382804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.382782    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-socket" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.385611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.385577    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-podinfo" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:29.386704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.386671    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-kube-api-access-85rh6" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "kube-api-access-85rh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:29.388140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.386872    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"]
Nov 03 08:21:29.388140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.387234    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-envoy" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.389352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.388439    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy"
Nov 03 08:21:29.389352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.388463    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy"
Nov 03 08:21:29.389352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.388756    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerName="istio-proxy"
Nov 03 08:21:29.389352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.389230    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-token" pod "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" (UID: "9f5abe5b-cb35-4119-a6c8-2c61b1b964b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:29.394565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.394095    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"]
Nov 03 08:21:29.394565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.394177    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.482772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.482731    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.482907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.482828    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.482907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.482870    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.483130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483052    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.483218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483155    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.483254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483230    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.483296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483251    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.483349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483333    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.483386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483377    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slgkk\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-kube-api-access-slgkk\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.483505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483490    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.483593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483513    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.483593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483528    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.483593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483542    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.483593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483565    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rh6\" (UniqueName: \"kubernetes.io/projected/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-kube-api-access-85rh6\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.483593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483579    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.483593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483592    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.483793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.483606    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.584896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.584853    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.584906    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgkk\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-kube-api-access-slgkk\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585080    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585161    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585201    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585305    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585337    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585417    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585453    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585526    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585555    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.585946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.585902    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.586158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.586139    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.586510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.586493    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.587493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.587474    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.587925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.587908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.593544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.593524    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.593631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.593547    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgkk\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-kube-api-access-slgkk\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.787313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.787221    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:29.941910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.941221    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 08:21:29.941910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.941277    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 08:21:29.960921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.944419    6430 generic.go:356] "Generic (PLEG): container finished" podID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" containerID="e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894" exitCode=0
Nov 03 08:21:29.960921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.944683    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" event={"ID":"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5","Type":"ContainerDied","Data":"e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894"}
Nov 03 08:21:29.960921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.944738    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4" event={"ID":"9f5abe5b-cb35-4119-a6c8-2c61b1b964b5","Type":"ContainerDied","Data":"93255979eef29fe9e6be3190e09e8ed390da46a8f9e4f9353c88e0eb9f15e309"}
Nov 03 08:21:29.960921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.944769    6430 scope.go:122] "RemoveContainer" containerID="e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894"
Nov 03 08:21:29.960921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.944928    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"
Nov 03 08:21:29.988929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.988841    6430 scope.go:122] "RemoveContainer" containerID="e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894"
Nov 03 08:21:29.989227 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:21:29.989144    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894\": container with ID starting with e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894 not found: ID does not exist" containerID="e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894"
Nov 03 08:21:29.989227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:29.989185    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894"} err="failed to get container status \"e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894\": rpc error: code = NotFound desc = could not find container \"e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894\": container with ID starting with e2da253d54c0362029aeb8dd220341c5d6fd0b9187ef0c593be3e0665c35e894 not found: ID does not exist"
Nov 03 08:21:30.038145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.033493    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:21:30.038145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.033556    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 08:21:30.059130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.053224    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"]
Nov 03 08:21:30.060351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.060325    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bxgs4"]
Nov 03 08:21:30.178817 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.178769    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:30.473123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.464215    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:30.473123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.464312    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:30.473123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.464358    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:30.473123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.467716    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"]
Nov 03 08:21:30.667668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.667639    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5abe5b-cb35-4119-a6c8-2c61b1b964b5" path="/var/lib/kubelet/pods/9f5abe5b-cb35-4119-a6c8-2c61b1b964b5/volumes"
Nov 03 08:21:30.949089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.942150    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 08:21:30.949089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:30.942215    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 08:21:31.061321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.061282    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" event={"ID":"a3df2b48-9c28-4c34-bde0-b7e35e55f41f","Type":"ContainerStarted","Data":"227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f"}
Nov 03 08:21:31.061912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.061887    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" event={"ID":"a3df2b48-9c28-4c34-bde0-b7e35e55f41f","Type":"ContainerStarted","Data":"6dabcba69c5466c67ba3996af8d4e4fb7e29c6574f1d806bb13120324a9b214e"}
Nov 03 08:21:31.070347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.070322    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:21:31.070535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.070509    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 08:21:31.096184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.096121    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podStartSLOduration=207.096103524 podStartE2EDuration="3m27.096103524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:31.091262404 +0000 UTC m=+1640.594097923" watchObservedRunningTime="2026-11-03 08:21:31.096103524 +0000 UTC m=+1640.598939041"
Nov 03 08:21:31.357486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.357384    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:31.696579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.695814    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:21:31.788033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.788008    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:31.788903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.788881    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:21:31.788987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.788930    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 08:21:31.939848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.939808    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 08:21:31.940177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:31.940140    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 08:21:32.032194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.032106    6430 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:21:32.032423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.032398    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 08:21:32.350720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.348711    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:32.695175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.695145    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:32.787995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.787956    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:21:32.788203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.788018    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 08:21:32.945295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.945216    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:32.945700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.945681    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:32.946349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:32.946333    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:21:33.043103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:33.042272    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:33.044415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:33.043370    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:33.047580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:33.047556    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:21:33.335967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:33.335861    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:33.788375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:33.788342    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:21:33.788785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:33.788394    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 08:21:33.869593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:33.869564    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:21:34.244297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:34.244257    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:34.788381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:34.788349    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:21:34.788772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:34.788403    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 08:21:35.096187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:35.088308    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:35.788161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:35.788124    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:21:35.788314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:35.788186    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 08:21:35.856223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:35.856184    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:36.656130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:36.654417    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:36.788566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:36.788535    6430 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:21:36.788743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:36.788592    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 08:21:37.412851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:37.412807    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:37.821234 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:37.821183    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:37.821929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:37.821876    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:37.824249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:37.824231    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:21:38.013838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:38.013805    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:38.150454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:38.150374    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:38.289048 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:38.289011    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:39.001040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:39.001000    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:39.739287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:39.739252    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:40.222893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:40.222864    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:40.342567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:40.342535    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:40.643881 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:40.643792    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:41.459900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:41.459864    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:42.259726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:42.259697    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:43.030380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:43.030350    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:43.286901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:43.286813    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"]
Nov 03 08:21:43.323329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:43.323276    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:43.971496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:43.971450    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:44.968651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:44.968614    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:46.046636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:46.046488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:46.757519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:46.757488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:47.554361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:47.554306    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:48.378298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:48.378269    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:49.285818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:49.285775    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:50.549213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:50.548210    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:51.619490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:51.619408    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:52.575691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:52.575659    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:52.711489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:52.711455    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:53.713292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:53.709441    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"]
Nov 03 08:21:53.713292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:53.709693    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j" podUID="088e6633-2419-4859-a7c3-cadcffb439d9" containerName="maas-api" containerID="cri-o://1997108fc3fd43ba6ec65162359a55763b80627f32ef17dd63bd2b60fb92e916" gracePeriod=30
Nov 03 08:21:53.818914 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:53.818874    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:54.022139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.017606    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:54.288139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.284948    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:54.456102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.456036    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:54.489648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.489612    6430 generic.go:356] "Generic (PLEG): container finished" podID="088e6633-2419-4859-a7c3-cadcffb439d9" containerID="1997108fc3fd43ba6ec65162359a55763b80627f32ef17dd63bd2b60fb92e916" exitCode=0
Nov 03 08:21:54.489821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.489740    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j" event={"ID":"088e6633-2419-4859-a7c3-cadcffb439d9","Type":"ContainerDied","Data":"1997108fc3fd43ba6ec65162359a55763b80627f32ef17dd63bd2b60fb92e916"}
Nov 03 08:21:54.489821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.489794    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j" event={"ID":"088e6633-2419-4859-a7c3-cadcffb439d9","Type":"ContainerDied","Data":"1e76a0a648d9316597d6b15a38c488ca881625043eb6ac68253911c9e971e246"}
Nov 03 08:21:54.489910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.489817    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e76a0a648d9316597d6b15a38c488ca881625043eb6ac68253911c9e971e246"
Nov 03 08:21:54.493014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.492935    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:54.549009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.544098    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"]
Nov 03 08:21:54.549009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.545419    6430 cpu_manager.go:414] "containerMap: removing container" podUID="088e6633-2419-4859-a7c3-cadcffb439d9" containerName="maas-api"
Nov 03 08:21:54.549009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.545440    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088e6633-2419-4859-a7c3-cadcffb439d9" containerName="maas-api"
Nov 03 08:21:54.549009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.545725    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="088e6633-2419-4859-a7c3-cadcffb439d9" containerName="maas-api"
Nov 03 08:21:54.550343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.550317    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.555482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.553710    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0d168e-serving-cert\""
Nov 03 08:21:54.572008 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.571663    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"]
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.583853    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088e6633-2419-4859-a7c3-cadcffb439d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088e6633-2419-4859-a7c3-cadcffb439d9-openshift-service-ca\") pod \"088e6633-2419-4859-a7c3-cadcffb439d9\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") "
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.584029    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-metrics-certs\") pod \"088e6633-2419-4859-a7c3-cadcffb439d9\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") "
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.584767    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088e6633-2419-4859-a7c3-cadcffb439d9-kube-api-access-tjvht\" (UniqueName: \"kubernetes.io/projected/088e6633-2419-4859-a7c3-cadcffb439d9-kube-api-access-tjvht\") pod \"088e6633-2419-4859-a7c3-cadcffb439d9\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") "
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.584837    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-maas-api-tls\") pod \"088e6633-2419-4859-a7c3-cadcffb439d9\" (UID: \"088e6633-2419-4859-a7c3-cadcffb439d9\") "
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.585047    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-metrics-certs\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.585107    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969cn\" (UniqueName: \"kubernetes.io/projected/542a29a3-ce27-4105-a01d-9ef6d4cb989f-kube-api-access-969cn\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.585143    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542a29a3-ce27-4105-a01d-9ef6d4cb989f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.585481    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.586047    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088e6633-2419-4859-a7c3-cadcffb439d9-openshift-service-ca" pod "088e6633-2419-4859-a7c3-cadcffb439d9" (UID: "088e6633-2419-4859-a7c3-cadcffb439d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.587226    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:21:54.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.587292    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:21:54.589214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.589176    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-metrics-certs" pod "088e6633-2419-4859-a7c3-cadcffb439d9" (UID: "088e6633-2419-4859-a7c3-cadcffb439d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:21:54.589551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.589524    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088e6633-2419-4859-a7c3-cadcffb439d9-kube-api-access-tjvht" pod "088e6633-2419-4859-a7c3-cadcffb439d9" (UID: "088e6633-2419-4859-a7c3-cadcffb439d9"). InnerVolumeSpecName "kube-api-access-tjvht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:54.593579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.591165    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-maas-api-tls" pod "088e6633-2419-4859-a7c3-cadcffb439d9" (UID: "088e6633-2419-4859-a7c3-cadcffb439d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:21:54.660943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.660907    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:54.687693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.687648    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.687892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.687831    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-metrics-certs\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.687892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.687866    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969cn\" (UniqueName: \"kubernetes.io/projected/542a29a3-ce27-4105-a01d-9ef6d4cb989f-kube-api-access-969cn\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.688022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.687915    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542a29a3-ce27-4105-a01d-9ef6d4cb989f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.688324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.688301    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:54.688437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.688331    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvht\" (UniqueName: \"kubernetes.io/projected/088e6633-2419-4859-a7c3-cadcffb439d9-kube-api-access-tjvht\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:54.688437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.688345    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088e6633-2419-4859-a7c3-cadcffb439d9-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:54.688437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.688360    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088e6633-2419-4859-a7c3-cadcffb439d9-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:54.690557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.690526    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542a29a3-ce27-4105-a01d-9ef6d4cb989f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.691994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.691966    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.693403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.693380    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-metrics-certs\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.700843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.700819    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-969cn\" (UniqueName: \"kubernetes.io/projected/542a29a3-ce27-4105-a01d-9ef6d4cb989f-kube-api-access-969cn\") pod \"maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.886266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.885699    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:54.974300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:54.974250    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:55.346738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:55.325783    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:55.455108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:55.455074    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:55.468610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:55.468568    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"]
Nov 03 08:21:55.619417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:55.619301    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"
Nov 03 08:21:55.668851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:55.668817    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"]
Nov 03 08:21:55.673532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:55.673502    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0c8a52-5c85699645-npq7j"]
Nov 03 08:21:56.237433 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.237400    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"]
Nov 03 08:21:56.260277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.260241    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"]
Nov 03 08:21:56.260452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.260349    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.266711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.266679    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a66d5c\"/\"openshift-service-ca.crt\""
Nov 03 08:21:56.266869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.266790    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a66d5c\"/\"kube-root-ca.crt\""
Nov 03 08:21:56.266971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.266953    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a66d5c\"/\"test-model-a66d5c-kserve-self-signed-certs\""
Nov 03 08:21:56.268325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.268302    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a66d5c\"/\"default-dockercfg-mkvvz\""
Nov 03 08:21:56.305157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.305124    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:56.337107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.336976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-home\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.337258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.337172    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e9de3e-9a77-4f01-980e-72497ced47b4-tls-certs\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.337331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.337282    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfm6\" (UniqueName: \"kubernetes.io/projected/42e9de3e-9a77-4f01-980e-72497ced47b4-kube-api-access-bjfm6\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.337390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.337336    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-model-cache\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.337390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.337371    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-dshm\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.337497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.337479    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-tmp-dir\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.468086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.455351    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfm6\" (UniqueName: \"kubernetes.io/projected/42e9de3e-9a77-4f01-980e-72497ced47b4-kube-api-access-bjfm6\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.468086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.455457    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-model-cache\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.468086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.455498    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-dshm\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.468086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.455634    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-tmp-dir\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.468086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.455790    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-home\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.468086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.455903    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e9de3e-9a77-4f01-980e-72497ced47b4-tls-certs\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.468086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.457253    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-model-cache\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.477085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.470720    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-home\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.492088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.486312    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-tmp-dir\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.553706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.536007    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfm6\" (UniqueName: \"kubernetes.io/projected/42e9de3e-9a77-4f01-980e-72497ced47b4-kube-api-access-bjfm6\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.553706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.538283    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-dshm\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.553706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.540623    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e9de3e-9a77-4f01-980e-72497ced47b4-tls-certs\") pod \"test-model-a66d5c-kserve-6d4956fcd9-6qm5q\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") " pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.594363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.593882    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:56.715087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.705451    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:56.824910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.824869    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088e6633-2419-4859-a7c3-cadcffb439d9" path="/var/lib/kubelet/pods/088e6633-2419-4859-a7c3-cadcffb439d9/volumes"
Nov 03 08:21:56.825385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.825365    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:21:56.825494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.825425    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr" event={"ID":"542a29a3-ce27-4105-a01d-9ef6d4cb989f","Type":"ContainerStarted","Data":"06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5"}
Nov 03 08:21:56.825494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.825455    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr" event={"ID":"542a29a3-ce27-4105-a01d-9ef6d4cb989f","Type":"ContainerStarted","Data":"baea0b80649602881af2e807763ff0e49d34838fd52381b98ab38d22de9845bc"}
Nov 03 08:21:56.844036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:56.843929    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr" podStartSLOduration=4.34021243 podStartE2EDuration="4.843906259s" totalImagesPullingTime="503.693829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:56.8392003 +0000 UTC m=+1666.342036117" watchObservedRunningTime="2026-11-03 08:21:56.843906259 +0000 UTC m=+1666.346741773"
Nov 03 08:21:57.171020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:57.170989    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"]
Nov 03 08:21:57.172215 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:21:57.172186    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42e9de3e_9a77_4f01_980e_72497ced47b4.slice/crio-05674532cc97cc66c250399a308ebf3f236f2affda1644a49a152715c76153e1 WatchSource:0}: Error finding container 05674532cc97cc66c250399a308ebf3f236f2affda1644a49a152715c76153e1: Status 404 returned error can't find the container with id 05674532cc97cc66c250399a308ebf3f236f2affda1644a49a152715c76153e1
Nov 03 08:21:57.583183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:57.583094    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:57.827992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:57.827952    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q" event={"ID":"42e9de3e-9a77-4f01-980e-72497ced47b4","Type":"ContainerStarted","Data":"1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9"}
Nov 03 08:21:57.830282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:57.828001    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q" event={"ID":"42e9de3e-9a77-4f01-980e-72497ced47b4","Type":"ContainerStarted","Data":"05674532cc97cc66c250399a308ebf3f236f2affda1644a49a152715c76153e1"}
Nov 03 08:21:58.321448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.321404    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" podUID="c62a3394-de42-4cda-b192-8157f006be53" containerName="main" containerID="cri-o://f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7" gracePeriod=45
Nov 03 08:21:58.377783 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:21:58.377740    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:21:58.529049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.529011    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:58.611767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.603212    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:21:58.739412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.739382    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:21:58.760581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.760533    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q" podStartSLOduration=2.760517215 podStartE2EDuration="2.760517215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:57.846130335 +0000 UTC m=+1667.348965850" watchObservedRunningTime="2026-11-03 08:21:58.760517215 +0000 UTC m=+1668.263352767"
Nov 03 08:21:58.832855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.832817    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-tmp-dir\") pod \"c62a3394-de42-4cda-b192-8157f006be53\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") "
Nov 03 08:21:58.833047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.832908    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-model-cache\") pod \"c62a3394-de42-4cda-b192-8157f006be53\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") "
Nov 03 08:21:58.833194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.833176    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-home\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-home\") pod \"c62a3394-de42-4cda-b192-8157f006be53\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") "
Nov 03 08:21:58.833250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.833186    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-model-cache" pod "c62a3394-de42-4cda-b192-8157f006be53" (UID: "c62a3394-de42-4cda-b192-8157f006be53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:58.833250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.833194    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-tmp-dir" pod "c62a3394-de42-4cda-b192-8157f006be53" (UID: "c62a3394-de42-4cda-b192-8157f006be53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:58.833250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.833231    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62a3394-de42-4cda-b192-8157f006be53-kube-api-access-m95sg\" (UniqueName: \"kubernetes.io/projected/c62a3394-de42-4cda-b192-8157f006be53-kube-api-access-m95sg\") pod \"c62a3394-de42-4cda-b192-8157f006be53\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") "
Nov 03 08:21:58.833396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.833319    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c62a3394-de42-4cda-b192-8157f006be53-tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a3394-de42-4cda-b192-8157f006be53-tls-certs\") pod \"c62a3394-de42-4cda-b192-8157f006be53\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") "
Nov 03 08:21:58.833504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.833433    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-home" pod "c62a3394-de42-4cda-b192-8157f006be53" (UID: "c62a3394-de42-4cda-b192-8157f006be53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:58.833504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.833461    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-dshm\") pod \"c62a3394-de42-4cda-b192-8157f006be53\" (UID: \"c62a3394-de42-4cda-b192-8157f006be53\") "
Nov 03 08:21:58.834267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.834238    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:58.834386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.834270    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:58.834386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.834285    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:58.835896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.835866    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-dshm" pod "c62a3394-de42-4cda-b192-8157f006be53" (UID: "c62a3394-de42-4cda-b192-8157f006be53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:58.836004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.835974    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62a3394-de42-4cda-b192-8157f006be53-kube-api-access-m95sg" pod "c62a3394-de42-4cda-b192-8157f006be53" (UID: "c62a3394-de42-4cda-b192-8157f006be53"). InnerVolumeSpecName "kube-api-access-m95sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:58.836312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.836289    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62a3394-de42-4cda-b192-8157f006be53-tls-certs" pod "c62a3394-de42-4cda-b192-8157f006be53" (UID: "c62a3394-de42-4cda-b192-8157f006be53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:21:58.840590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.840559    6430 generic.go:356] "Generic (PLEG): container finished" podID="c62a3394-de42-4cda-b192-8157f006be53" containerID="f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7" exitCode=0
Nov 03 08:21:58.840685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.840659    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"
Nov 03 08:21:58.840731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.840698    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" event={"ID":"c62a3394-de42-4cda-b192-8157f006be53","Type":"ContainerDied","Data":"f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7"}
Nov 03 08:21:58.840786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.840748    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln" event={"ID":"c62a3394-de42-4cda-b192-8157f006be53","Type":"ContainerDied","Data":"6f09b1faf327f4b795a92e9775476cb2c275088020d18233b72126503f61bac5"}
Nov 03 08:21:58.840786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.840771    6430 scope.go:122] "RemoveContainer" containerID="f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7"
Nov 03 08:21:58.843638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.843140    6430 generic.go:356] "Generic (PLEG): container finished" podID="b04237f6-0845-4c53-b888-1f0898562832" containerID="c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32" exitCode=0
Nov 03 08:21:58.843638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.843187    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" event={"ID":"b04237f6-0845-4c53-b888-1f0898562832","Type":"ContainerDied","Data":"c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32"}
Nov 03 08:21:58.843801 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.843744    6430 scope.go:122] "RemoveContainer" containerID="c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32"
Nov 03 08:21:58.847708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.847655    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:58.847807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.847754    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:58.847851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.847809    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:21:58.881877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.881854    6430 scope.go:122] "RemoveContainer" containerID="f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7"
Nov 03 08:21:58.882242 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:21:58.882219    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7\": container with ID starting with f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7 not found: ID does not exist" containerID="f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7"
Nov 03 08:21:58.882340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.882252    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7"} err="failed to get container status \"f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7\": rpc error: code = NotFound desc = could not find container \"f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7\": container with ID starting with f5cc65931118aa4941c6b8c71cc1d331b0f4890066113da2c2f75c02d7f79ab7 not found: ID does not exist"
Nov 03 08:21:58.917917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.917869    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"]
Nov 03 08:21:58.922389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.922347    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ba69a0/test-model-ba69a0-kserve-d5b5475cc-zw5ln"]
Nov 03 08:21:58.936450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.936047    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62a3394-de42-4cda-b192-8157f006be53-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:58.936450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.936131    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95sg\" (UniqueName: \"kubernetes.io/projected/c62a3394-de42-4cda-b192-8157f006be53-kube-api-access-m95sg\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:58.936450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:58.936149    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62a3394-de42-4cda-b192-8157f006be53-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:59.282546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:59.282501    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:21:59.862168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:21:59.862109    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" event={"ID":"b04237f6-0845-4c53-b888-1f0898562832","Type":"ContainerStarted","Data":"27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1"}
Nov 03 08:22:00.084516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:00.084481    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:00.337259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:00.337231    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:22:00.337432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:00.337381    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 08:22:00.337432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:00.337417    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 08:22:00.634191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:00.634115    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62a3394-de42-4cda-b192-8157f006be53" path="/var/lib/kubelet/pods/c62a3394-de42-4cda-b192-8157f006be53/volumes"
Nov 03 08:22:00.977642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:00.977611    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:01.337475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:01.337397    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 08:22:01.337475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:01.337446    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 08:22:01.450987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:01.450948    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:01.620138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:01.619932    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:01.779268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:01.779217    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:02.032108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.031656    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:02.415134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.393266    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 08:22:02.415134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.393345    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 08:22:02.458250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.458217    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"]
Nov 03 08:22:02.458537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.458499    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy" containerID="cri-o://68516de526dea87710aff4b3bf9ab4c8c384385883384204eaab05494785dc84" gracePeriod=30
Nov 03 08:22:02.850102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.847378    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"]
Nov 03 08:22:02.850102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.847680    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy" containerID="cri-o://84ae452b147a5098d19318f68bf3787236331801de39235baab15fc8f96ba1f7" gracePeriod=30
Nov 03 08:22:02.857206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:02.857170    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:22:03.274744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:03.274711    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:03.346000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:03.345964    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 08:22:03.346296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:03.346267    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 08:22:04.337899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:04.337862    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 08:22:04.338342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:04.337928    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 08:22:04.409087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:04.408477    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:04.907016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:04.906979    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:05.099091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:05.086260    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"]
Nov 03 08:22:05.360687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:05.360653    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 08:22:05.361310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:05.361277    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 08:22:05.862309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:05.862259    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:06.412645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:06.412610    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:22:06.413188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:06.413097    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:22:06.414689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:06.414663    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:22:06.594940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:06.594914    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:22:06.608752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:06.608724    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:22:06.743655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:06.743615    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:07.083260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:07.083190    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:22:07.680290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:07.680254    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:08.084522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.084308    6430 generic.go:356] "Generic (PLEG): container finished" podID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerID="84ae452b147a5098d19318f68bf3787236331801de39235baab15fc8f96ba1f7" exitCode=0
Nov 03 08:22:08.084522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.084448    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" event={"ID":"72d91edf-d0c3-42fa-85b5-7d76d72a562c","Type":"ContainerDied","Data":"84ae452b147a5098d19318f68bf3787236331801de39235baab15fc8f96ba1f7"}
Nov 03 08:22:08.087832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.086501    6430 generic.go:356] "Generic (PLEG): container finished" podID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerID="68516de526dea87710aff4b3bf9ab4c8c384385883384204eaab05494785dc84" exitCode=0
Nov 03 08:22:08.087832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.087243    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" event={"ID":"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe","Type":"ContainerDied","Data":"68516de526dea87710aff4b3bf9ab4c8c384385883384204eaab05494785dc84"}
Nov 03 08:22:08.227936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.227746    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:22:08.285727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.285691    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-token\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-token\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.285884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.285773    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istiod-ca-cert\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.285884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.285842    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-certs\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.286387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.285999    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-kube-api-access-d8prf\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-kube-api-access-d8prf\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.286387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.286049    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-credential-socket\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.286387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.286207    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-socket\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.286387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.286240    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-podinfo\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.286387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.286373    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-data\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.286679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.286423    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-envoy\") pod \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\" (UID: \"72d91edf-d0c3-42fa-85b5-7d76d72a562c\") "
Nov 03 08:22:08.289035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.288873    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istiod-ca-cert" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:08.289461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.289436    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-socket" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.290209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.289968    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-certs" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.290209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.290173    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-data" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.290540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.290514    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-credential-socket" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.294480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.294005    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-envoy" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.294480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.294203    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-token" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.294480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.294261    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-kube-api-access-d8prf" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "kube-api-access-d8prf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.295881 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.295850    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-podinfo" pod "72d91edf-d0c3-42fa-85b5-7d76d72a562c" (UID: "72d91edf-d0c3-42fa-85b5-7d76d72a562c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:22:08.299024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.298999    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:22:08.388787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.388753    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-kube-api-access-4pn98\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-kube-api-access-4pn98\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.388992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.388819    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-envoy\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.389093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.388998    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-credential-socket\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.389167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.389133    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-podinfo\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.389347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.389306    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-credential-socket" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.389499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.389166    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-token\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-token\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.389630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.389610    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-socket\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.389888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.389862    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-socket" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.390052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.390030    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-data\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.390209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.390160    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istiod-ca-cert\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.390278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.390231    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-certs\") pod \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\" (UID: \"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe\") "
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.390852    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-certs" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.390900    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istiod-ca-cert" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.390924    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-data" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391389    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391415    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8prf\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-kube-api-access-d8prf\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391430    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391445    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391460    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391474    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391486    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391499    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391512    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391523    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.391619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.391536    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.393141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.392394    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.393141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.392424    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d91edf-d0c3-42fa-85b5-7d76d72a562c-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.393141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.392439    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d91edf-d0c3-42fa-85b5-7d76d72a562c-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.394391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.394344    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-envoy" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.394391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.394353    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-podinfo" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:22:08.394546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.394485    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-kube-api-access-4pn98" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "kube-api-access-4pn98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.394696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.394675    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-token" pod "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" (UID: "4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.494014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.493978    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pn98\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-kube-api-access-4pn98\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.494014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.494017    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.494242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.494034    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.494242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.494045    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.546533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.546485    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:08.815871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.815788    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"]
Nov 03 08:22:08.816806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.816787    6430 cpu_manager.go:414] "containerMap: removing container" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy"
Nov 03 08:22:08.816806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.816807    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy"
Nov 03 08:22:08.816964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.816874    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c62a3394-de42-4cda-b192-8157f006be53" containerName="main"
Nov 03 08:22:08.816964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.816879    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62a3394-de42-4cda-b192-8157f006be53" containerName="main"
Nov 03 08:22:08.816964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.816890    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy"
Nov 03 08:22:08.816964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.816894    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy"
Nov 03 08:22:08.817208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.817020    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" containerName="istio-proxy"
Nov 03 08:22:08.817208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.817036    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62a3394-de42-4cda-b192-8157f006be53" containerName="main"
Nov 03 08:22:08.817208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.817043    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" containerName="istio-proxy"
Nov 03 08:22:08.831013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.829901    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:08.837079 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.836867    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"]
Nov 03 08:22:08.837927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.837898    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-109070\"/\"test-model-109070-kserve-self-signed-certs\""
Nov 03 08:22:08.838092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.838051    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-109070\"/\"openshift-service-ca.crt\""
Nov 03 08:22:08.838179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.838163    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-109070\"/\"kube-root-ca.crt\""
Nov 03 08:22:08.838304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.838290    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-109070\"/\"default-dockercfg-frnj5\""
Nov 03 08:22:08.885442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.885410    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:08.903595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.903567    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-model-cache\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:08.903748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.903713    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-dshm\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:08.903799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.903776    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7c09ad-494e-412e-9d98-093a3e5df344-tls-certs\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:08.903851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.903819    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-tmp-dir\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:08.904020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.903992    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq6v\" (UniqueName: \"kubernetes.io/projected/1b7c09ad-494e-412e-9d98-093a3e5df344-kube-api-access-qqq6v\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:08.904228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:08.904207    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-home\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.010089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.009402    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7c09ad-494e-412e-9d98-093a3e5df344-tls-certs\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.010089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.009470    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-tmp-dir\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.010089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.009572    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq6v\" (UniqueName: \"kubernetes.io/projected/1b7c09ad-494e-412e-9d98-093a3e5df344-kube-api-access-qqq6v\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.010089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.009751    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-home\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.010089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.009879    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-model-cache\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.010089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.009981    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-dshm\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.038084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.026182    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-home\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.038084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.027385    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-tmp-dir\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.038084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.027617    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-model-cache\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.038084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.032989    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7c09ad-494e-412e-9d98-093a3e5df344-tls-certs\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.059099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.043749    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-dshm\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.059099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.048780    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq6v\" (UniqueName: \"kubernetes.io/projected/1b7c09ad-494e-412e-9d98-093a3e5df344-kube-api-access-qqq6v\") pod \"test-model-109070-kserve-cc466b949-5v2wg\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") " pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.146147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.137558    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst" event={"ID":"72d91edf-d0c3-42fa-85b5-7d76d72a562c","Type":"ContainerDied","Data":"d2cf9a9c352b6e1fb5162c1b09fd1d77adfe9e3afeb281c8216dd14690327a09"}
Nov 03 08:22:09.146147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.137630    6430 scope.go:122] "RemoveContainer" containerID="84ae452b147a5098d19318f68bf3787236331801de39235baab15fc8f96ba1f7"
Nov 03 08:22:09.146147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.137898    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"
Nov 03 08:22:09.159300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.159265    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:09.191093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.182863    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m" event={"ID":"4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe","Type":"ContainerDied","Data":"69ee7a9d1722034b1d03f5595ef56f470757f3ea143a7c9af780ab0602fafdb6"}
Nov 03 08:22:09.191093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.183140    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"
Nov 03 08:22:09.211152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.211119    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"]
Nov 03 08:22:09.245419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.245385    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-7cxst"]
Nov 03 08:22:09.340835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.338721    6430 scope.go:122] "RemoveContainer" containerID="68516de526dea87710aff4b3bf9ab4c8c384385883384204eaab05494785dc84"
Nov 03 08:22:09.351286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.351257    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"]
Nov 03 08:22:09.358535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.358510    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-jpf4m"]
Nov 03 08:22:09.477502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.477464    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:09.617089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:09.616049    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"]
Nov 03 08:22:10.194052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.194018    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg" event={"ID":"1b7c09ad-494e-412e-9d98-093a3e5df344","Type":"ContainerStarted","Data":"139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc"}
Nov 03 08:22:10.194557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.194113    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg" event={"ID":"1b7c09ad-494e-412e-9d98-093a3e5df344","Type":"ContainerStarted","Data":"6dfee1425e508143068d9bb81b590429f2cf829a865abc0dbb7b57ab9a24a0c9"}
Nov 03 08:22:10.218836 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.218779    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg" podStartSLOduration=2.21876093 podStartE2EDuration="2.21876093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:10.212354374 +0000 UTC m=+1679.715189889" watchObservedRunningTime="2026-11-03 08:22:10.21876093 +0000 UTC m=+1679.721596452"
Nov 03 08:22:10.374194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.374164    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:10.621741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.621665    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe" path="/var/lib/kubelet/pods/4dd2af55-fc63-4e95-9a95-6c2a2e8fdcfe/volumes"
Nov 03 08:22:10.622175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.622158    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d91edf-d0c3-42fa-85b5-7d76d72a562c" path="/var/lib/kubelet/pods/72d91edf-d0c3-42fa-85b5-7d76d72a562c/volumes"
Nov 03 08:22:10.979111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.971717    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"]
Nov 03 08:22:10.979111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:10.971937    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5" podUID="85ff3ca9-1574-4328-9ec3-d942c5941e1a" containerName="maas-api" containerID="cri-o://90cf093368e314bd44db85d744247f64dda82d0b7c11fa86cff6d92ca2cf935f" gracePeriod=30
Nov 03 08:22:11.057125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.046919    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:11.166900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.162137    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:11.450194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.420506    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:11.557582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.557545    6430 generic.go:356] "Generic (PLEG): container finished" podID="85ff3ca9-1574-4328-9ec3-d942c5941e1a" containerID="90cf093368e314bd44db85d744247f64dda82d0b7c11fa86cff6d92ca2cf935f" exitCode=0
Nov 03 08:22:11.578636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.578110    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5" event={"ID":"85ff3ca9-1574-4328-9ec3-d942c5941e1a","Type":"ContainerDied","Data":"90cf093368e314bd44db85d744247f64dda82d0b7c11fa86cff6d92ca2cf935f"}
Nov 03 08:22:11.578636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.578533    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:11.731913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.731886    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:11.789093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.787378    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:22:11.825154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.825119    6430 scope.go:122] "RemoveContainer" containerID="371c029c2651a6c9d9dff6b049744c4e36ef7885f0cb291241a3502a31a6c1c4"
Nov 03 08:22:11.872734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.870917    6430 scope.go:122] "RemoveContainer" containerID="7f0f92c51a6ecc7bc24058cca14aafc8f646684010fae68503fad33c98216bbd"
Nov 03 08:22:11.910323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.906165    6430 scope.go:122] "RemoveContainer" containerID="35771ccde7d9220d4ccdeb0df04c3717744eb7e52a39f317ebdf1e93376b7cfb"
Nov 03 08:22:11.955012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.954976    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:11.975682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.975652    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-maas-api-tls\") pod \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") "
Nov 03 08:22:11.976012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.975991    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ff3ca9-1574-4328-9ec3-d942c5941e1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ff3ca9-1574-4328-9ec3-d942c5941e1a-openshift-service-ca\") pod \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") "
Nov 03 08:22:11.976407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.976381    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ff3ca9-1574-4328-9ec3-d942c5941e1a-openshift-service-ca" pod "85ff3ca9-1574-4328-9ec3-d942c5941e1a" (UID: "85ff3ca9-1574-4328-9ec3-d942c5941e1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:11.978767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.978674    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ff3ca9-1574-4328-9ec3-d942c5941e1a-kube-api-access-7zslr\" (UniqueName: \"kubernetes.io/projected/85ff3ca9-1574-4328-9ec3-d942c5941e1a-kube-api-access-7zslr\") pod \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") "
Nov 03 08:22:11.978767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.978721    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-metrics-certs\") pod \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\" (UID: \"85ff3ca9-1574-4328-9ec3-d942c5941e1a\") "
Nov 03 08:22:11.979711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.979603    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ff3ca9-1574-4328-9ec3-d942c5941e1a-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:11.980547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.980519    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-maas-api-tls" pod "85ff3ca9-1574-4328-9ec3-d942c5941e1a" (UID: "85ff3ca9-1574-4328-9ec3-d942c5941e1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:11.985190 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.984705    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-metrics-certs" pod "85ff3ca9-1574-4328-9ec3-d942c5941e1a" (UID: "85ff3ca9-1574-4328-9ec3-d942c5941e1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:11.985190 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:11.984769    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ff3ca9-1574-4328-9ec3-d942c5941e1a-kube-api-access-7zslr" pod "85ff3ca9-1574-4328-9ec3-d942c5941e1a" (UID: "85ff3ca9-1574-4328-9ec3-d942c5941e1a"). InnerVolumeSpecName "kube-api-access-7zslr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:12.085075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.083225    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zslr\" (UniqueName: \"kubernetes.io/projected/85ff3ca9-1574-4328-9ec3-d942c5941e1a-kube-api-access-7zslr\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:12.085075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.083343    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:12.085075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.083365    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ff3ca9-1574-4328-9ec3-d942c5941e1a-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:12.567637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.567605    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"
Nov 03 08:22:12.567637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.567604    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5" event={"ID":"85ff3ca9-1574-4328-9ec3-d942c5941e1a","Type":"ContainerDied","Data":"f6413f9c4bb47efb99c53bfbbbc9588a7c4768c270f252eb3d74eb240ff6fb6e"}
Nov 03 08:22:12.568177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.567708    6430 scope.go:122] "RemoveContainer" containerID="90cf093368e314bd44db85d744247f64dda82d0b7c11fa86cff6d92ca2cf935f"
Nov 03 08:22:12.590921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.590891    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"]
Nov 03 08:22:12.596899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.596873    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f6f94-76b9b558ff-wlsz5"]
Nov 03 08:22:12.619682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.619652    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ff3ca9-1574-4328-9ec3-d942c5941e1a" path="/var/lib/kubelet/pods/85ff3ca9-1574-4328-9ec3-d942c5941e1a/volumes"
Nov 03 08:22:12.775200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:12.775169    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:13.535325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:13.535295    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:13.835019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:13.834935    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"]
Nov 03 08:22:13.846304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:13.846270    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"]
Nov 03 08:22:13.909354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:13.909313    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:14.033391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:14.033340    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:14.640558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:14.640520    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:15.311422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:15.311392    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:15.435521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:15.435486    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:15.594258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:15.590984    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:16.564260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:16.564229    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:17.314974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:17.314941    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:18.246301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.246269    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:18.483630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.483584    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:18.616177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.616097    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:18.771892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.771850    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"]
Nov 03 08:22:18.772188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.772157    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" containerID="cri-o://27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1" gracePeriod=30
Nov 03 08:22:18.826362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.826322    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:18.835288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.835255    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w" podUID="a1c424eb-6899-4156-a468-21a2b95f28bd" containerName="payload-processing" containerID="cri-o://4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83" gracePeriod=25
Nov 03 08:22:18.846741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:18.846706    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8" podUID="5a521420-dbc3-4449-91c1-af058118cfb9" containerName="payload-pre-processing" containerID="cri-o://a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3" gracePeriod=25
Nov 03 08:22:19.162881 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.160435    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:19.267592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.257431    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:19.498238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.498206    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:22:19.504570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.504504    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:22:19.612562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.612526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:19.660704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.646867    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a521420-dbc3-4449-91c1-af058118cfb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a521420-dbc3-4449-91c1-af058118cfb9-plugins-config-volume\") pod \"5a521420-dbc3-4449-91c1-af058118cfb9\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") "
Nov 03 08:22:19.661271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.661242    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c424eb-6899-4156-a468-21a2b95f28bd-kube-api-access-zjkzt\" (UniqueName: \"kubernetes.io/projected/a1c424eb-6899-4156-a468-21a2b95f28bd-kube-api-access-zjkzt\") pod \"a1c424eb-6899-4156-a468-21a2b95f28bd\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") "
Nov 03 08:22:19.661612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.660337    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a521420-dbc3-4449-91c1-af058118cfb9-plugins-config-volume" pod "5a521420-dbc3-4449-91c1-af058118cfb9" (UID: "5a521420-dbc3-4449-91c1-af058118cfb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:19.661612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.661602    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a521420-dbc3-4449-91c1-af058118cfb9-kube-api-access-27sz5\" (UniqueName: \"kubernetes.io/projected/5a521420-dbc3-4449-91c1-af058118cfb9-kube-api-access-27sz5\") pod \"5a521420-dbc3-4449-91c1-af058118cfb9\" (UID: \"5a521420-dbc3-4449-91c1-af058118cfb9\") "
Nov 03 08:22:19.661778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.661648    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c424eb-6899-4156-a468-21a2b95f28bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c424eb-6899-4156-a468-21a2b95f28bd-plugins-config-volume\") pod \"a1c424eb-6899-4156-a468-21a2b95f28bd\" (UID: \"a1c424eb-6899-4156-a468-21a2b95f28bd\") "
Nov 03 08:22:19.675087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.669280    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c424eb-6899-4156-a468-21a2b95f28bd-plugins-config-volume" pod "a1c424eb-6899-4156-a468-21a2b95f28bd" (UID: "a1c424eb-6899-4156-a468-21a2b95f28bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:19.675877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.675843    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a521420-dbc3-4449-91c1-af058118cfb9-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:19.676153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.676134    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c424eb-6899-4156-a468-21a2b95f28bd-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:19.676773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.676731    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c424eb-6899-4156-a468-21a2b95f28bd-kube-api-access-zjkzt" pod "a1c424eb-6899-4156-a468-21a2b95f28bd" (UID: "a1c424eb-6899-4156-a468-21a2b95f28bd"). InnerVolumeSpecName "kube-api-access-zjkzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:19.686436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.686392    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a521420-dbc3-4449-91c1-af058118cfb9-kube-api-access-27sz5" pod "5a521420-dbc3-4449-91c1-af058118cfb9" (UID: "5a521420-dbc3-4449-91c1-af058118cfb9"). InnerVolumeSpecName "kube-api-access-27sz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:19.756541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.756458    6430 generic.go:356] "Generic (PLEG): container finished" podID="5a521420-dbc3-4449-91c1-af058118cfb9" containerID="a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3" exitCode=0
Nov 03 08:22:19.756702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.756567    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"
Nov 03 08:22:19.756702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.756596    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8" event={"ID":"5a521420-dbc3-4449-91c1-af058118cfb9","Type":"ContainerDied","Data":"a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3"}
Nov 03 08:22:19.756702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.756653    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8" event={"ID":"5a521420-dbc3-4449-91c1-af058118cfb9","Type":"ContainerDied","Data":"3e21dfdf0aef7b9e1618354f86a5616e70456b0f1323812aba2273319cc6f94d"}
Nov 03 08:22:19.756702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.756687    6430 scope.go:122] "RemoveContainer" containerID="a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3"
Nov 03 08:22:19.758448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.758279    6430 generic.go:356] "Generic (PLEG): container finished" podID="a1c424eb-6899-4156-a468-21a2b95f28bd" containerID="4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83" exitCode=0
Nov 03 08:22:19.758448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.758321    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"
Nov 03 08:22:19.758448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.758438    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w" event={"ID":"a1c424eb-6899-4156-a468-21a2b95f28bd","Type":"ContainerDied","Data":"4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83"}
Nov 03 08:22:19.758702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.758488    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w" event={"ID":"a1c424eb-6899-4156-a468-21a2b95f28bd","Type":"ContainerDied","Data":"8f40e0e2d0e9c20d8a0a9a2b1064316e8f23a74b2291a360a4d51830782f0d69"}
Nov 03 08:22:19.770303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.770253    6430 scope.go:122] "RemoveContainer" containerID="a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3"
Nov 03 08:22:19.770528 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:19.770510    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3\": container with ID starting with a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3 not found: ID does not exist" containerID="a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3"
Nov 03 08:22:19.770587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.770536    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3"} err="failed to get container status \"a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3\": rpc error: code = NotFound desc = could not find container \"a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3\": container with ID starting with a129be9adab1b9e86bfb736305059bdbef3398d1de0743bea5140a36d3a74bd3 not found: ID does not exist"
Nov 03 08:22:19.770587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.770557    6430 scope.go:122] "RemoveContainer" containerID="4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83"
Nov 03 08:22:19.777685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.777667    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:22:19.778409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.778385    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjkzt\" (UniqueName: \"kubernetes.io/projected/a1c424eb-6899-4156-a468-21a2b95f28bd-kube-api-access-zjkzt\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:19.778409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.778413    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27sz5\" (UniqueName: \"kubernetes.io/projected/5a521420-dbc3-4449-91c1-af058118cfb9-kube-api-access-27sz5\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:19.785674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.785596    6430 scope.go:122] "RemoveContainer" containerID="4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83"
Nov 03 08:22:19.785674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.785619    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"]
Nov 03 08:22:19.785934 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:19.785906    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83\": container with ID starting with 4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83 not found: ID does not exist" containerID="4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83"
Nov 03 08:22:19.786020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.785943    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83"} err="failed to get container status \"4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83\": rpc error: code = NotFound desc = could not find container \"4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83\": container with ID starting with 4e627430fc53580412d44e7b1061ae8ef06945d0f3d016388556b2ba2fb96f83 not found: ID does not exist"
Nov 03 08:22:19.790537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.790501    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a6481-74656868ff-2wk5w"]
Nov 03 08:22:19.824126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.824099    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"]
Nov 03 08:22:19.832903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:19.830606    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a6481-66bfcb7459-jjbx8"]
Nov 03 08:22:20.332567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:20.332525    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk" podUID="10f80321-1867-4c5d-8462-24a3a17c4831" containerName="main" containerID="cri-o://16714577501c69c98d0d3d7fb6bc43dac33b043d5737ae152b0df344d252866a" gracePeriod=45
Nov 03 08:22:20.702254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:20.702222    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a521420-dbc3-4449-91c1-af058118cfb9" path="/var/lib/kubelet/pods/5a521420-dbc3-4449-91c1-af058118cfb9/volumes"
Nov 03 08:22:20.702756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:20.702710    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c424eb-6899-4156-a468-21a2b95f28bd" path="/var/lib/kubelet/pods/a1c424eb-6899-4156-a468-21a2b95f28bd/volumes"
Nov 03 08:22:20.787582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:20.787541    6430 generic.go:356] "Generic (PLEG): container finished" podID="10f80321-1867-4c5d-8462-24a3a17c4831" containerID="16714577501c69c98d0d3d7fb6bc43dac33b043d5737ae152b0df344d252866a" exitCode=0
Nov 03 08:22:20.787749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:20.787642    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk" event={"ID":"10f80321-1867-4c5d-8462-24a3a17c4831","Type":"ContainerDied","Data":"16714577501c69c98d0d3d7fb6bc43dac33b043d5737ae152b0df344d252866a"}
Nov 03 08:22:20.950376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:20.949581    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:20.955207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:20.954612    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:22:21.096424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.096394    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-dshm\") pod \"10f80321-1867-4c5d-8462-24a3a17c4831\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") "
Nov 03 08:22:21.096613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.096488    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-tmp-dir\") pod \"10f80321-1867-4c5d-8462-24a3a17c4831\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") "
Nov 03 08:22:21.096613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.096563    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f80321-1867-4c5d-8462-24a3a17c4831-kube-api-access-pntrr\" (UniqueName: \"kubernetes.io/projected/10f80321-1867-4c5d-8462-24a3a17c4831-kube-api-access-pntrr\") pod \"10f80321-1867-4c5d-8462-24a3a17c4831\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") "
Nov 03 08:22:21.096613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.096603    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-home\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-home\") pod \"10f80321-1867-4c5d-8462-24a3a17c4831\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") "
Nov 03 08:22:21.096787 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.096727    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-tmp-dir" pod "10f80321-1867-4c5d-8462-24a3a17c4831" (UID: "10f80321-1867-4c5d-8462-24a3a17c4831"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:21.096840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.096784    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-model-cache\") pod \"10f80321-1867-4c5d-8462-24a3a17c4831\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") "
Nov 03 08:22:21.096924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.096905    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-home" pod "10f80321-1867-4c5d-8462-24a3a17c4831" (UID: "10f80321-1867-4c5d-8462-24a3a17c4831"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:21.097047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.097030    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f80321-1867-4c5d-8462-24a3a17c4831-tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80321-1867-4c5d-8462-24a3a17c4831-tls-certs\") pod \"10f80321-1867-4c5d-8462-24a3a17c4831\" (UID: \"10f80321-1867-4c5d-8462-24a3a17c4831\") "
Nov 03 08:22:21.097354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.097029    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-model-cache" pod "10f80321-1867-4c5d-8462-24a3a17c4831" (UID: "10f80321-1867-4c5d-8462-24a3a17c4831"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:21.097745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.097725    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:21.097822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.097750    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:21.097822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.097763    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:21.099031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.099001    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f80321-1867-4c5d-8462-24a3a17c4831-kube-api-access-pntrr" pod "10f80321-1867-4c5d-8462-24a3a17c4831" (UID: "10f80321-1867-4c5d-8462-24a3a17c4831"). InnerVolumeSpecName "kube-api-access-pntrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:21.099288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.099266    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-dshm" pod "10f80321-1867-4c5d-8462-24a3a17c4831" (UID: "10f80321-1867-4c5d-8462-24a3a17c4831"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:21.099387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.099373    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f80321-1867-4c5d-8462-24a3a17c4831-tls-certs" pod "10f80321-1867-4c5d-8462-24a3a17c4831" (UID: "10f80321-1867-4c5d-8462-24a3a17c4831"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:21.199908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.199876    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f80321-1867-4c5d-8462-24a3a17c4831-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:21.199908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.199903    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f80321-1867-4c5d-8462-24a3a17c4831-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:21.199908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.199913    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pntrr\" (UniqueName: \"kubernetes.io/projected/10f80321-1867-4c5d-8462-24a3a17c4831-kube-api-access-pntrr\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:21.415448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.415373    6430 patch_prober.go:29] interesting pod/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:22:21.415448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.415440    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:22:21.796827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.796801    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"
Nov 03 08:22:21.797027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.796820    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk" event={"ID":"10f80321-1867-4c5d-8462-24a3a17c4831","Type":"ContainerDied","Data":"72c60b65c420d7a19d6b1681e5bb998b300bf3e5313bf9e51dc9e509a951f355"}
Nov 03 08:22:21.797027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.796875    6430 scope.go:122] "RemoveContainer" containerID="16714577501c69c98d0d3d7fb6bc43dac33b043d5737ae152b0df344d252866a"
Nov 03 08:22:21.819350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.819313    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"]
Nov 03 08:22:21.823444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.823422    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-179c16/test-model-179c16-kserve-55c658c8df-zz4dk"]
Nov 03 08:22:21.845296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:21.845274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:22.419359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:22.419326    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:22.621186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:22.621153    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f80321-1867-4c5d-8462-24a3a17c4831" path="/var/lib/kubelet/pods/10f80321-1867-4c5d-8462-24a3a17c4831/volumes"
Nov 03 08:22:22.627452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:22.627428    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:23.564082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:23.563562    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:24.082390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.082363    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:22:24.176831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.176750    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-data\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.176831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.176800    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-kube-api-access-m5bgw\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-kube-api-access-m5bgw\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.177107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.176904    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-istio-token\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-istio-token\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.177107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.176944    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04237f6-0845-4c53-b888-1f0898562832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04237f6-0845-4c53-b888-1f0898562832-istiod-ca-cert\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.177107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.176989    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-credential-socket\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.177107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.177017    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b04237f6-0845-4c53-b888-1f0898562832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04237f6-0845-4c53-b888-1f0898562832-istio-podinfo\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.177107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.177055    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-socket\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.177987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.177149    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-envoy\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.177987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.177145    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-data" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:24.177987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.177387    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-credential-socket" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:24.177987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.177626    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04237f6-0845-4c53-b888-1f0898562832-istiod-ca-cert" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:24.177987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.177694    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-certs\") pod \"b04237f6-0845-4c53-b888-1f0898562832\" (UID: \"b04237f6-0845-4c53-b888-1f0898562832\") "
Nov 03 08:22:24.178378 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.178342    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-socket" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:24.178718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.178676    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-certs" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:24.178859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.178830    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04237f6-0845-4c53-b888-1f0898562832-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.178859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.178856    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.178990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.178870    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.178990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.178885    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.178990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.178898    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.179778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.179682    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-envoy" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:24.180274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.180256    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b04237f6-0845-4c53-b888-1f0898562832-istio-podinfo" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:22:24.180849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.180828    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-istio-token" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:24.181303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.181281    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-kube-api-access-m5bgw" pod "b04237f6-0845-4c53-b888-1f0898562832" (UID: "b04237f6-0845-4c53-b888-1f0898562832"). InnerVolumeSpecName "kube-api-access-m5bgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:24.228367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.228330    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"]
Nov 03 08:22:24.229254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229226    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy"
Nov 03 08:22:24.229254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229248    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229262    6430 cpu_manager.go:414] "containerMap: removing container" podUID="85ff3ca9-1574-4328-9ec3-d942c5941e1a" containerName="maas-api"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229270    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ff3ca9-1574-4328-9ec3-d942c5941e1a" containerName="maas-api"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229294    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a1c424eb-6899-4156-a468-21a2b95f28bd" containerName="payload-processing"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229302    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c424eb-6899-4156-a468-21a2b95f28bd" containerName="payload-processing"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229314    6430 cpu_manager.go:414] "containerMap: removing container" podUID="10f80321-1867-4c5d-8462-24a3a17c4831" containerName="main"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229320    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f80321-1867-4c5d-8462-24a3a17c4831" containerName="main"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229352    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5a521420-dbc3-4449-91c1-af058118cfb9" containerName="payload-pre-processing"
Nov 03 08:22:24.229496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229359    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a521420-dbc3-4449-91c1-af058118cfb9" containerName="payload-pre-processing"
Nov 03 08:22:24.229887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229572    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy"
Nov 03 08:22:24.229887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229587    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ff3ca9-1574-4328-9ec3-d942c5941e1a" containerName="maas-api"
Nov 03 08:22:24.229887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229601    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c424eb-6899-4156-a468-21a2b95f28bd" containerName="payload-processing"
Nov 03 08:22:24.229887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229612    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f80321-1867-4c5d-8462-24a3a17c4831" containerName="main"
Nov 03 08:22:24.229887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229622    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy"
Nov 03 08:22:24.229887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.229636    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a521420-dbc3-4449-91c1-af058118cfb9" containerName="payload-pre-processing"
Nov 03 08:22:24.234987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.234964    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.246040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.245228    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0fc79b\"/\"openshift-service-ca.crt\""
Nov 03 08:22:24.246040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.245271    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0fc79b\"/\"test-model-0fc79b-kserve-self-signed-certs\""
Nov 03 08:22:24.246040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.245377    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0fc79b\"/\"default-dockercfg-8wq4p\""
Nov 03 08:22:24.246362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.246267    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"]
Nov 03 08:22:24.267506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.267475    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:24.280752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.280713    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2xx\" (UniqueName: \"kubernetes.io/projected/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-kube-api-access-bf2xx\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.281020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.280995    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-model-cache\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.281135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281038    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tls-certs\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.281135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281089    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-home\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.281253 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tmp-dir\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.281253 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281212    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-dshm\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.281427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281411    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04237f6-0845-4c53-b888-1f0898562832-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.281491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281432    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04237f6-0845-4c53-b888-1f0898562832-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.281491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281448    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5bgw\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-kube-api-access-m5bgw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.281491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.281464    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04237f6-0845-4c53-b888-1f0898562832-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.341943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.341905    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0fc79b\"/\"kube-root-ca.crt\""
Nov 03 08:22:24.393101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.386334    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-home\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.393101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.386422    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tmp-dir\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.393101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.386559    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-dshm\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.393101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.386770    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2xx\" (UniqueName: \"kubernetes.io/projected/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-kube-api-access-bf2xx\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.393101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.387181    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-model-cache\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.393101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.387211    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tls-certs\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.397171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.396263    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tmp-dir\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.397171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.396825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-model-cache\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.416448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.413251    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-home\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.440135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.426936    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:24.440135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.433184    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2xx\" (UniqueName: \"kubernetes.io/projected/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-kube-api-access-bf2xx\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.442204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.442138    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tls-certs\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.442619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.442575    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-dshm\") pod \"test-model-0fc79b-kserve-79ffc49dcc-tqjk4\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") " pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.571089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.570381    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:24.829482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.829448    6430 generic.go:356] "Generic (PLEG): container finished" podID="b04237f6-0845-4c53-b888-1f0898562832" containerID="27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1" exitCode=0
Nov 03 08:22:24.829732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.829712    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" event={"ID":"b04237f6-0845-4c53-b888-1f0898562832","Type":"ContainerDied","Data":"27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1"}
Nov 03 08:22:24.829822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.829769    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj" event={"ID":"b04237f6-0845-4c53-b888-1f0898562832","Type":"ContainerDied","Data":"b520f12578635fe7ce79fa6e22b5f51a40b6a75adaaf5bdfeef57cd6bc8d9280"}
Nov 03 08:22:24.829822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.829797    6430 scope.go:122] "RemoveContainer" containerID="27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1"
Nov 03 08:22:24.829950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.829937    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"
Nov 03 08:22:24.851644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.851619    6430 scope.go:122] "RemoveContainer" containerID="c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32"
Nov 03 08:22:24.853249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.852912    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"]
Nov 03 08:22:24.859866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.859838    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7f6f94-openshift-default-6b58c575df-g6qdj"]
Nov 03 08:22:24.873859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.873835    6430 scope.go:122] "RemoveContainer" containerID="27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1"
Nov 03 08:22:24.874241 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:24.874212    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1\": container with ID starting with 27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1 not found: ID does not exist" containerID="27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1"
Nov 03 08:22:24.874343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.874257    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1"} err="failed to get container status \"27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1\": rpc error: code = NotFound desc = could not find container \"27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1\": container with ID starting with 27c1c96f7e2f2a23d1cfdc1133c6bfd2242ac967bfe2c52ef548824f4735bcc1 not found: ID does not exist"
Nov 03 08:22:24.874343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.874284    6430 scope.go:122] "RemoveContainer" containerID="c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32"
Nov 03 08:22:24.874611 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:24.874583    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32\": container with ID starting with c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32 not found: ID does not exist" containerID="c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32"
Nov 03 08:22:24.874689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.874622    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32"} err="failed to get container status \"c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32\": rpc error: code = NotFound desc = could not find container \"c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32\": container with ID starting with c882a0041c34e77d0a299a94912451a28a3e299ccb3ad851de739c065e878e32 not found: ID does not exist"
Nov 03 08:22:24.992087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:24.985411    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"]
Nov 03 08:22:25.046688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:25.046658    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:25.840408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:25.840373    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4" event={"ID":"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b","Type":"ContainerStarted","Data":"69f30cc0a7b40576c8f797602fcfea54fde22260195584bb664bc9f0098761a4"}
Nov 03 08:22:25.840408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:25.840415    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4" event={"ID":"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b","Type":"ContainerStarted","Data":"3216b6bf61837c424750cde45040c502971e0a7becdf7eae41aa3dc1a9634cf3"}
Nov 03 08:22:25.845869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:25.845842    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:26.571740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:26.571707    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:26.621493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:26.621444    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04237f6-0845-4c53-b888-1f0898562832" path="/var/lib/kubelet/pods/b04237f6-0845-4c53-b888-1f0898562832/volumes"
Nov 03 08:22:26.655438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:26.655414    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:27.234479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:27.234437    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:27.437089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:27.427492    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:27.764086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:27.761777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:27.920580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:27.920550    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:28.377833 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:28.377798    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:22:28.411271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:28.411239    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:28.965122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:28.965047    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:29.600548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:29.600492    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4" podStartSLOduration=5.600476415 podStartE2EDuration="5.600476415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:25.868781397 +0000 UTC m=+1695.371616912" watchObservedRunningTime="2026-11-03 08:22:29.600476415 +0000 UTC m=+1699.103311929"
Nov 03 08:22:29.602567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:29.602545    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"]
Nov 03 08:22:29.615134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:29.615103    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"]
Nov 03 08:22:29.707810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:29.707766    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:29.836942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:29.836851    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:30.526965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:30.526934    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:30.720720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:30.720685    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:30.852806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:30.852725    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:31.175363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:31.175245    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:31.311798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:31.311770    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:32.261307 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:32.261276    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:33.347135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:33.347038    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:34.150256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.150224    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:34.512551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.512502    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:34.585085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.574882    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:34.603147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.603084    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj" podUID="47e8aa56-70f1-4832-b52d-89a3ee495556" containerName="payload-processing" containerID="cri-o://ea4c27824be282b6315325ddba52b1928da19f6810e48e2fe761f4f3c9f42a1b" gracePeriod=25
Nov 03 08:22:34.605738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.605710    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:34.616268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.616236    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2" podUID="47341871-69c7-411a-ab2e-8463095626e0" containerName="payload-pre-processing" containerID="cri-o://73a25e38f7d5476fe5ccf4c0f7f915fed39bd5d932e3eede5a856a558abf5cd7" gracePeriod=25
Nov 03 08:22:34.629111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.629085    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:34.791384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.791290    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:35.047133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.998617    6430 generic.go:356] "Generic (PLEG): container finished" podID="47e8aa56-70f1-4832-b52d-89a3ee495556" containerID="ea4c27824be282b6315325ddba52b1928da19f6810e48e2fe761f4f3c9f42a1b" exitCode=0
Nov 03 08:22:35.047133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:34.999649    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj" event={"ID":"47e8aa56-70f1-4832-b52d-89a3ee495556","Type":"ContainerDied","Data":"ea4c27824be282b6315325ddba52b1928da19f6810e48e2fe761f4f3c9f42a1b"}
Nov 03 08:22:35.047133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.002836    6430 generic.go:356] "Generic (PLEG): container finished" podID="47341871-69c7-411a-ab2e-8463095626e0" containerID="73a25e38f7d5476fe5ccf4c0f7f915fed39bd5d932e3eede5a856a558abf5cd7" exitCode=0
Nov 03 08:22:35.047133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.007106    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2" event={"ID":"47341871-69c7-411a-ab2e-8463095626e0","Type":"ContainerDied","Data":"73a25e38f7d5476fe5ccf4c0f7f915fed39bd5d932e3eede5a856a558abf5cd7"}
Nov 03 08:22:35.047133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.027931    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:22:35.123085 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:22:35.117256    6430 logging.go:55] [core] [Channel #3898 SubChannel #3900]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.81:50052", ServerName: "10.133.0.81:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.81:50052: connect: connection refused"
Nov 03 08:22:35.254439 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.254411    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:22:35.310684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.310608    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:35.313999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.313974    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:22:35.359196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.348510    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47341871-69c7-411a-ab2e-8463095626e0-kube-api-access-vglkf\" (UniqueName: \"kubernetes.io/projected/47341871-69c7-411a-ab2e-8463095626e0-kube-api-access-vglkf\") pod \"47341871-69c7-411a-ab2e-8463095626e0\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") "
Nov 03 08:22:35.359196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.348636    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47341871-69c7-411a-ab2e-8463095626e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47341871-69c7-411a-ab2e-8463095626e0-plugins-config-volume\") pod \"47341871-69c7-411a-ab2e-8463095626e0\" (UID: \"47341871-69c7-411a-ab2e-8463095626e0\") "
Nov 03 08:22:35.359196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.348694    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e8aa56-70f1-4832-b52d-89a3ee495556-kube-api-access-4rvjc\" (UniqueName: \"kubernetes.io/projected/47e8aa56-70f1-4832-b52d-89a3ee495556-kube-api-access-4rvjc\") pod \"47e8aa56-70f1-4832-b52d-89a3ee495556\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") "
Nov 03 08:22:35.359196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.348824    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e8aa56-70f1-4832-b52d-89a3ee495556-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e8aa56-70f1-4832-b52d-89a3ee495556-plugins-config-volume\") pod \"47e8aa56-70f1-4832-b52d-89a3ee495556\" (UID: \"47e8aa56-70f1-4832-b52d-89a3ee495556\") "
Nov 03 08:22:35.359196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.350249    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e8aa56-70f1-4832-b52d-89a3ee495556-plugins-config-volume" pod "47e8aa56-70f1-4832-b52d-89a3ee495556" (UID: "47e8aa56-70f1-4832-b52d-89a3ee495556"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:35.371138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.370381    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47341871-69c7-411a-ab2e-8463095626e0-plugins-config-volume" pod "47341871-69c7-411a-ab2e-8463095626e0" (UID: "47341871-69c7-411a-ab2e-8463095626e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:35.380246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.377789    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e8aa56-70f1-4832-b52d-89a3ee495556-kube-api-access-4rvjc" pod "47e8aa56-70f1-4832-b52d-89a3ee495556" (UID: "47e8aa56-70f1-4832-b52d-89a3ee495556"). InnerVolumeSpecName "kube-api-access-4rvjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:35.451724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.451690    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rvjc\" (UniqueName: \"kubernetes.io/projected/47e8aa56-70f1-4832-b52d-89a3ee495556-kube-api-access-4rvjc\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:35.451724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.451725    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e8aa56-70f1-4832-b52d-89a3ee495556-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:35.452035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.451742    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47341871-69c7-411a-ab2e-8463095626e0-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:35.453607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.453573    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47341871-69c7-411a-ab2e-8463095626e0-kube-api-access-vglkf" pod "47341871-69c7-411a-ab2e-8463095626e0" (UID: "47341871-69c7-411a-ab2e-8463095626e0"). InnerVolumeSpecName "kube-api-access-vglkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:35.552920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:35.552892    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vglkf\" (UniqueName: \"kubernetes.io/projected/47341871-69c7-411a-ab2e-8463095626e0-kube-api-access-vglkf\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:36.013053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.013024    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"
Nov 03 08:22:36.013260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.013025    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj" event={"ID":"47e8aa56-70f1-4832-b52d-89a3ee495556","Type":"ContainerDied","Data":"12f247f58a7070a13e6e358fb62ef460c3097343223d5043e179f8ecbb69c9e5"}
Nov 03 08:22:36.013260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.013145    6430 scope.go:122] "RemoveContainer" containerID="ea4c27824be282b6315325ddba52b1928da19f6810e48e2fe761f4f3c9f42a1b"
Nov 03 08:22:36.014749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.014727    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"
Nov 03 08:22:36.014897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.014761    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2" event={"ID":"47341871-69c7-411a-ab2e-8463095626e0","Type":"ContainerDied","Data":"c0266df3a1f7ea63ba677bc76799d894d4da187065c4967e801d6bfdb06eefae"}
Nov 03 08:22:36.025602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.025585    6430 scope.go:122] "RemoveContainer" containerID="73a25e38f7d5476fe5ccf4c0f7f915fed39bd5d932e3eede5a856a558abf5cd7"
Nov 03 08:22:36.042367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.042338    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"]
Nov 03 08:22:36.046739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.046716    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj"]
Nov 03 08:22:36.056455 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.056433    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"]
Nov 03 08:22:36.063036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.063015    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2"]
Nov 03 08:22:36.074209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.074184    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a60004-5775cbbf5-4d7wj" podUID="47e8aa56-70f1-4832-b52d-89a3ee495556" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.82:50052\" within 1s: context deadline exceeded"
Nov 03 08:22:36.074329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.074253    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a60004-5b9dccd5cf-dx9h2" podUID="47341871-69c7-411a-ab2e-8463095626e0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.81:50052\" within 1s: context deadline exceeded"
Nov 03 08:22:36.074329 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:22:36.074266    6430 logging.go:55] [core] [Channel #3897 SubChannel #3899]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.82:50052", ServerName: "10.133.0.82:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.82:50052: operation was canceled"
Nov 03 08:22:36.335367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.335256    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:36.620328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.620257    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47341871-69c7-411a-ab2e-8463095626e0" path="/var/lib/kubelet/pods/47341871-69c7-411a-ab2e-8463095626e0/volumes"
Nov 03 08:22:36.620821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:36.620806    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e8aa56-70f1-4832-b52d-89a3ee495556" path="/var/lib/kubelet/pods/47e8aa56-70f1-4832-b52d-89a3ee495556/volumes"
Nov 03 08:22:37.192685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.192645    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:37.247754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.247637    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"]
Nov 03 08:22:37.249210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.249181    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy"
Nov 03 08:22:37.249377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.249364    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04237f6-0845-4c53-b888-1f0898562832" containerName="istio-proxy"
Nov 03 08:22:37.249527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.249516    6430 cpu_manager.go:414] "containerMap: removing container" podUID="47341871-69c7-411a-ab2e-8463095626e0" containerName="payload-pre-processing"
Nov 03 08:22:37.249602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.249592    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47341871-69c7-411a-ab2e-8463095626e0" containerName="payload-pre-processing"
Nov 03 08:22:37.249721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.249710    6430 cpu_manager.go:414] "containerMap: removing container" podUID="47e8aa56-70f1-4832-b52d-89a3ee495556" containerName="payload-processing"
Nov 03 08:22:37.249801 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.249791    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e8aa56-70f1-4832-b52d-89a3ee495556" containerName="payload-processing"
Nov 03 08:22:37.250175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.250155    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e8aa56-70f1-4832-b52d-89a3ee495556" containerName="payload-processing"
Nov 03 08:22:37.250318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.250306    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="47341871-69c7-411a-ab2e-8463095626e0" containerName="payload-pre-processing"
Nov 03 08:22:37.271305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.268924    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.284707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.283507    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d168e\"/\"openshift-service-ca.crt\""
Nov 03 08:22:37.284707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.283640    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d168e\"/\"default-dockercfg-nnk2q\""
Nov 03 08:22:37.284707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.283721    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d168e\"/\"kube-root-ca.crt\""
Nov 03 08:22:37.284707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.284425    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d168e\"/\"test-model-0d168e-kserve-self-signed-certs\""
Nov 03 08:22:37.288972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.288931    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"]
Nov 03 08:22:37.290288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.289925    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tls-certs\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.290288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.290148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tmp-dir\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.290288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.290182    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djs5d\" (UniqueName: \"kubernetes.io/projected/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-kube-api-access-djs5d\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.290288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.290217    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-model-cache\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.290288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.290271    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-home\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.290820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.290344    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-dshm\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.320116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.319332    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:37.391827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.391799    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tmp-dir\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.391992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.391834    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djs5d\" (UniqueName: \"kubernetes.io/projected/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-kube-api-access-djs5d\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.391992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.391862    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-model-cache\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.391992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.391915    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-home\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.392218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.392194    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-dshm\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.392314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.392293    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tmp-dir\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.392410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.392389    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-home\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.392410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.392401    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tls-certs\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.392536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.392432    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-model-cache\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.394833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.394811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-dshm\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.395375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.395351    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tls-certs\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.398722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.398695    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djs5d\" (UniqueName: \"kubernetes.io/projected/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-kube-api-access-djs5d\") pod \"test-model-0d168e-kserve-59575f75-59frr\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") " pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.615138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.614328    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:37.953975 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:37.953940    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:38.098365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:38.098342    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"]
Nov 03 08:22:38.100143 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:22:38.100079    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff2495b_3061_4d9f_a53f_78e4f9ff2adc.slice/crio-547c17ada101967e5c2b048e1ca64336b996124aaadec7adb4557c78a24a7c82 WatchSource:0}: Error finding container 547c17ada101967e5c2b048e1ca64336b996124aaadec7adb4557c78a24a7c82: Status 404 returned error can't find the container with id 547c17ada101967e5c2b048e1ca64336b996124aaadec7adb4557c78a24a7c82
Nov 03 08:22:38.582407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:38.582373    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:38.710462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:38.710426    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:38.951274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:38.951236    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:39.142351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:39.142245    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:39.155363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:39.155325    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr" event={"ID":"eff2495b-3061-4d9f-a53f-78e4f9ff2adc","Type":"ContainerStarted","Data":"6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329"}
Nov 03 08:22:39.155646 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:39.155621    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr" event={"ID":"eff2495b-3061-4d9f-a53f-78e4f9ff2adc","Type":"ContainerStarted","Data":"547c17ada101967e5c2b048e1ca64336b996124aaadec7adb4557c78a24a7c82"}
Nov 03 08:22:39.184104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:39.184029    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr" podStartSLOduration=2.184010282 podStartE2EDuration="2.184010282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:39.183362542 +0000 UTC m=+1708.686198060" watchObservedRunningTime="2026-11-03 08:22:39.184010282 +0000 UTC m=+1708.686845796"
Nov 03 08:22:39.614919 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:39.614885    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:40.322976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:40.322949    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:41.461238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:41.461199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:43.130097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:43.130030    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q" podUID="42e9de3e-9a77-4f01-980e-72497ced47b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:43.352087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:43.340601    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:44.349100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:44.348250    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:45.271689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:45.271656    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:45.488790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:45.488752    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"]
Nov 03 08:22:45.535094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:45.515531    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"]
Nov 03 08:22:45.605410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:45.605372    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:45.716493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:45.716450    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:46.457396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:46.457363    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:47.482305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:47.482268    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:47.614937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:47.614904    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:47.629346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:47.629307    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:48.261954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:48.261921    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:48.402912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:48.402889    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:22:49.145239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:49.145169    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:49.593116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:49.593081    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"]
Nov 03 08:22:49.689417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:49.689384    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:49.802293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:49.802263    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:50.221214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.221149    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:50.490112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.489989    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks" podUID="be030d14-64df-4b28-8224-145b22b369a5" containerName="payload-processing" containerID="cri-o://3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75" gracePeriod=25
Nov 03 08:22:50.516724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.516693    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw" podUID="ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" containerName="payload-pre-processing" containerID="cri-o://b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24" gracePeriod=25
Nov 03 08:22:50.834191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.834169    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:22:50.853540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.853504    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:22:50.963596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.963559    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be030d14-64df-4b28-8224-145b22b369a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be030d14-64df-4b28-8224-145b22b369a5-plugins-config-volume\") pod \"be030d14-64df-4b28-8224-145b22b369a5\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") "
Nov 03 08:22:50.963790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.963660    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-plugins-config-volume\") pod \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") "
Nov 03 08:22:50.963860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.963841    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be030d14-64df-4b28-8224-145b22b369a5-kube-api-access-llqtn\" (UniqueName: \"kubernetes.io/projected/be030d14-64df-4b28-8224-145b22b369a5-kube-api-access-llqtn\") pod \"be030d14-64df-4b28-8224-145b22b369a5\" (UID: \"be030d14-64df-4b28-8224-145b22b369a5\") "
Nov 03 08:22:50.963917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.963890    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be030d14-64df-4b28-8224-145b22b369a5-plugins-config-volume" pod "be030d14-64df-4b28-8224-145b22b369a5" (UID: "be030d14-64df-4b28-8224-145b22b369a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:50.963980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.963941    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-kube-api-access-tb2lm\" (UniqueName: \"kubernetes.io/projected/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-kube-api-access-tb2lm\") pod \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\" (UID: \"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b\") "
Nov 03 08:22:50.964112 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.964050    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-plugins-config-volume" pod "ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" (UID: "ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:50.964684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.964664    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:50.964745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.964686    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be030d14-64df-4b28-8224-145b22b369a5-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:50.966161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.966136    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be030d14-64df-4b28-8224-145b22b369a5-kube-api-access-llqtn" pod "be030d14-64df-4b28-8224-145b22b369a5" (UID: "be030d14-64df-4b28-8224-145b22b369a5"). InnerVolumeSpecName "kube-api-access-llqtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:50.966247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:50.966175    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-kube-api-access-tb2lm" pod "ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" (UID: "ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b"). InnerVolumeSpecName "kube-api-access-tb2lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:51.065887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.065808    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llqtn\" (UniqueName: \"kubernetes.io/projected/be030d14-64df-4b28-8224-145b22b369a5-kube-api-access-llqtn\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:51.065887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.065835    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2lm\" (UniqueName: \"kubernetes.io/projected/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b-kube-api-access-tb2lm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:51.149720 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.149688    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:51.414725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.414630    6430 generic.go:356] "Generic (PLEG): container finished" podID="be030d14-64df-4b28-8224-145b22b369a5" containerID="3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75" exitCode=0
Nov 03 08:22:51.414725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.414679    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"
Nov 03 08:22:51.415265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.414716    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks" event={"ID":"be030d14-64df-4b28-8224-145b22b369a5","Type":"ContainerDied","Data":"3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75"}
Nov 03 08:22:51.415265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.414777    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks" event={"ID":"be030d14-64df-4b28-8224-145b22b369a5","Type":"ContainerDied","Data":"2d5e524c3ba81394ca8bf66c9ce584f34e97f66bd49287921f6271d446db8746"}
Nov 03 08:22:51.415265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.414809    6430 scope.go:122] "RemoveContainer" containerID="3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75"
Nov 03 08:22:51.416466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.416431    6430 generic.go:356] "Generic (PLEG): container finished" podID="ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" containerID="b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24" exitCode=0
Nov 03 08:22:51.416601 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.416461    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw" event={"ID":"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b","Type":"ContainerDied","Data":"b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24"}
Nov 03 08:22:51.416601 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.416519    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw" event={"ID":"ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b","Type":"ContainerDied","Data":"df38018cf85ac5cf4ec7acf34e7874e4e9274afe04719b8dd4d03dec7640be74"}
Nov 03 08:22:51.416601 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.416475    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"
Nov 03 08:22:51.426262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.426246    6430 scope.go:122] "RemoveContainer" containerID="3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75"
Nov 03 08:22:51.426508 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:51.426491    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75\": container with ID starting with 3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75 not found: ID does not exist" containerID="3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75"
Nov 03 08:22:51.426580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.426516    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75"} err="failed to get container status \"3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75\": rpc error: code = NotFound desc = could not find container \"3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75\": container with ID starting with 3fff63c3987b8989e14f1efaae2803b1fb89072f7e9512e01d3a06077f6f2e75 not found: ID does not exist"
Nov 03 08:22:51.426580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.426535    6430 scope.go:122] "RemoveContainer" containerID="b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24"
Nov 03 08:22:51.436750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.436734    6430 scope.go:122] "RemoveContainer" containerID="b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24"
Nov 03 08:22:51.437011 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:51.436985    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24\": container with ID starting with b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24 not found: ID does not exist" containerID="b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24"
Nov 03 08:22:51.437157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.437019    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24"} err="failed to get container status \"b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24\": rpc error: code = NotFound desc = could not find container \"b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24\": container with ID starting with b975dba4a5897591ee9e7fb6ba51f160bcfa984056b2012a0a3350626b64bc24 not found: ID does not exist"
Nov 03 08:22:51.442409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.442389    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"]
Nov 03 08:22:51.446223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.446201    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e81ace-7d9bd7577-lc9ks"]
Nov 03 08:22:51.451712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.451690    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"]
Nov 03 08:22:51.454899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.454881    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e81ace-578749fd75-9fzfw"]
Nov 03 08:22:51.928683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:51.928650    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:52.621620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:52.621590    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be030d14-64df-4b28-8224-145b22b369a5" path="/var/lib/kubelet/pods/be030d14-64df-4b28-8224-145b22b369a5/volumes"
Nov 03 08:22:52.622087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:52.622054    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" path="/var/lib/kubelet/pods/ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b/volumes"
Nov 03 08:22:52.856093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:52.855275    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:53.345717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:53.345590    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"]
Nov 03 08:22:53.345923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:53.345854    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd" podUID="9ec8f62e-72ce-485b-868e-c142e9d14291" containerName="maas-api" containerID="cri-o://0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4" gracePeriod=30
Nov 03 08:22:53.425848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:53.425814    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:53.878120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:53.863901    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:54.046088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.044493    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:54.175912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.175879    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:22:54.298512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.298480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec8f62e-72ce-485b-868e-c142e9d14291-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec8f62e-72ce-485b-868e-c142e9d14291-openshift-service-ca\") pod \"9ec8f62e-72ce-485b-868e-c142e9d14291\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") "
Nov 03 08:22:54.298689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.298621    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec8f62e-72ce-485b-868e-c142e9d14291-kube-api-access-wvbx2\" (UniqueName: \"kubernetes.io/projected/9ec8f62e-72ce-485b-868e-c142e9d14291-kube-api-access-wvbx2\") pod \"9ec8f62e-72ce-485b-868e-c142e9d14291\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") "
Nov 03 08:22:54.298689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.298651    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls\") pod \"9ec8f62e-72ce-485b-868e-c142e9d14291\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") "
Nov 03 08:22:54.298830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.298815    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-metrics-certs\") pod \"9ec8f62e-72ce-485b-868e-c142e9d14291\" (UID: \"9ec8f62e-72ce-485b-868e-c142e9d14291\") "
Nov 03 08:22:54.298872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.298844    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec8f62e-72ce-485b-868e-c142e9d14291-openshift-service-ca" pod "9ec8f62e-72ce-485b-868e-c142e9d14291" (UID: "9ec8f62e-72ce-485b-868e-c142e9d14291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:54.299587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.299564    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec8f62e-72ce-485b-868e-c142e9d14291-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:54.300888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.300857    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls" pod "9ec8f62e-72ce-485b-868e-c142e9d14291" (UID: "9ec8f62e-72ce-485b-868e-c142e9d14291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:54.301111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.301092    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-metrics-certs" pod "9ec8f62e-72ce-485b-868e-c142e9d14291" (UID: "9ec8f62e-72ce-485b-868e-c142e9d14291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:54.301193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.301129    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec8f62e-72ce-485b-868e-c142e9d14291-kube-api-access-wvbx2" pod "9ec8f62e-72ce-485b-868e-c142e9d14291" (UID: "9ec8f62e-72ce-485b-868e-c142e9d14291"). InnerVolumeSpecName "kube-api-access-wvbx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:54.317725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.317690    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:54.401443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.401365    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvbx2\" (UniqueName: \"kubernetes.io/projected/9ec8f62e-72ce-485b-868e-c142e9d14291-kube-api-access-wvbx2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:54.401443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.401395    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:54.401443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.401404    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec8f62e-72ce-485b-868e-c142e9d14291-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:54.444045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.444014    6430 generic.go:356] "Generic (PLEG): container finished" podID="9ec8f62e-72ce-485b-868e-c142e9d14291" containerID="0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4" exitCode=0
Nov 03 08:22:54.444168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.444054    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd" event={"ID":"9ec8f62e-72ce-485b-868e-c142e9d14291","Type":"ContainerDied","Data":"0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4"}
Nov 03 08:22:54.444168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.444098    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"
Nov 03 08:22:54.444168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.444129    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd" event={"ID":"9ec8f62e-72ce-485b-868e-c142e9d14291","Type":"ContainerDied","Data":"25c480d0643d5656ca816fd14d63701d88ced1d2d419d2e5da68a7fd0c92a94a"}
Nov 03 08:22:54.444168 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.444150    6430 scope.go:122] "RemoveContainer" containerID="0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4"
Nov 03 08:22:54.455608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.455590    6430 scope.go:122] "RemoveContainer" containerID="0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4"
Nov 03 08:22:54.455869 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:54.455846    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4\": container with ID starting with 0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4 not found: ID does not exist" containerID="0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4"
Nov 03 08:22:54.455935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.455879    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4"} err="failed to get container status \"0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4\": rpc error: code = NotFound desc = could not find container \"0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4\": container with ID starting with 0e5481abaed1a955eb5a5be31217e08a4b061ac6f5fd1f2299fdb4aae944ebb4 not found: ID does not exist"
Nov 03 08:22:54.466610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.466589    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"]
Nov 03 08:22:54.475508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.474807    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a60004-97cddc4f5-9snrd"]
Nov 03 08:22:54.593967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.593933    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" podUID="475aa1d3-6841-4350-959b-85d2ac5517bf" containerName="payload-pre-processing" containerID="cri-o://843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd" gracePeriod=25
Nov 03 08:22:54.620109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.620043    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec8f62e-72ce-485b-868e-c142e9d14291" path="/var/lib/kubelet/pods/9ec8f62e-72ce-485b-868e-c142e9d14291/volumes"
Nov 03 08:22:54.919301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:54.919267    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:55.049654 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:55.049622    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:55.172902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:55.172825    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:55.742388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:55.742356    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:56.522756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:56.522715    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:57.576859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:57.571995    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:58.377378 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:22:58.377334    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:22:58.460453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:58.460407    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:22:59.333938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:22:59.333901    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:00.064119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:00.064045    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" podUID="475aa1d3-6841-4350-959b-85d2ac5517bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:00.250685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:00.250649    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:00.662693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:00.662649    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:00.783382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:00.783351    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:00.983091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:00.983028    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:01.451189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:01.451156    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:02.587389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:02.587349    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:03.544185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:03.544148    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:04.829413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:04.829371    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"]
Nov 03 08:23:04.829946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:04.829645    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4" podUID="fcadc141-2f8d-4474-81cd-9aeac02c02aa" containerName="maas-api" containerID="cri-o://e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf" gracePeriod=30
Nov 03 08:23:05.390114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.332639    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:05.613605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.612824    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:05.682683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.682659    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:23:05.687370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.687342    6430 generic.go:356] "Generic (PLEG): container finished" podID="fcadc141-2f8d-4474-81cd-9aeac02c02aa" containerID="e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf" exitCode=0
Nov 03 08:23:05.687519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.687388    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"
Nov 03 08:23:05.687519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.687404    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4" event={"ID":"fcadc141-2f8d-4474-81cd-9aeac02c02aa","Type":"ContainerDied","Data":"e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf"}
Nov 03 08:23:05.687519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.687438    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4" event={"ID":"fcadc141-2f8d-4474-81cd-9aeac02c02aa","Type":"ContainerDied","Data":"dd3a0a603b215174e4ad9bf643756b2dbdcfeebd2b4f9f9acb3b514053a539b5"}
Nov 03 08:23:05.687519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.687459    6430 scope.go:122] "RemoveContainer" containerID="e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf"
Nov 03 08:23:05.697942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.697920    6430 scope.go:122] "RemoveContainer" containerID="e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf"
Nov 03 08:23:05.698235 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:05.698215    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf\": container with ID starting with e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf not found: ID does not exist" containerID="e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf"
Nov 03 08:23:05.698293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.698244    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf"} err="failed to get container status \"e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf\": rpc error: code = NotFound desc = could not find container \"e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf\": container with ID starting with e9361331ec3c54c01764f2e164c16e4b2759ea239d09307b03add3b3e300d8bf not found: ID does not exist"
Nov 03 08:23:05.731214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.731187    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-metrics-certs\") pod \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") "
Nov 03 08:23:05.731379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.731259    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcadc141-2f8d-4474-81cd-9aeac02c02aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcadc141-2f8d-4474-81cd-9aeac02c02aa-openshift-service-ca\") pod \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") "
Nov 03 08:23:05.731379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.731356    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-maas-api-tls\") pod \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") "
Nov 03 08:23:05.731887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.731857    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcadc141-2f8d-4474-81cd-9aeac02c02aa-openshift-service-ca" pod "fcadc141-2f8d-4474-81cd-9aeac02c02aa" (UID: "fcadc141-2f8d-4474-81cd-9aeac02c02aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:05.732090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.732050    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcadc141-2f8d-4474-81cd-9aeac02c02aa-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:05.733696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.733664    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-metrics-certs" pod "fcadc141-2f8d-4474-81cd-9aeac02c02aa" (UID: "fcadc141-2f8d-4474-81cd-9aeac02c02aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:05.734146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.734038    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-maas-api-tls" pod "fcadc141-2f8d-4474-81cd-9aeac02c02aa" (UID: "fcadc141-2f8d-4474-81cd-9aeac02c02aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:05.832971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.832938    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcadc141-2f8d-4474-81cd-9aeac02c02aa-kube-api-access-26htz\" (UniqueName: \"kubernetes.io/projected/fcadc141-2f8d-4474-81cd-9aeac02c02aa-kube-api-access-26htz\") pod \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\" (UID: \"fcadc141-2f8d-4474-81cd-9aeac02c02aa\") "
Nov 03 08:23:05.833495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.833470    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:05.833637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.833498    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcadc141-2f8d-4474-81cd-9aeac02c02aa-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:05.835352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.835325    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcadc141-2f8d-4474-81cd-9aeac02c02aa-kube-api-access-26htz" pod "fcadc141-2f8d-4474-81cd-9aeac02c02aa" (UID: "fcadc141-2f8d-4474-81cd-9aeac02c02aa"). InnerVolumeSpecName "kube-api-access-26htz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:05.935054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:05.935018    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26htz\" (UniqueName: \"kubernetes.io/projected/fcadc141-2f8d-4474-81cd-9aeac02c02aa-kube-api-access-26htz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.009832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.009805    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"]
Nov 03 08:23:06.016731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.016703    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e81ace-56657bd754-5jxh4"]
Nov 03 08:23:06.230230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.230137    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"]
Nov 03 08:23:06.230417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.230392    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl" podUID="b7dda9a6-e738-403b-8b3a-2aa22356b419" containerName="maas-api" containerID="cri-o://5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f" gracePeriod=30
Nov 03 08:23:06.351528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.351481    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:06.636704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.624888    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:06.654089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.648687    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcadc141-2f8d-4474-81cd-9aeac02c02aa" path="/var/lib/kubelet/pods/fcadc141-2f8d-4474-81cd-9aeac02c02aa/volumes"
Nov 03 08:23:06.708105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.698962    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:23:06.758828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.758798    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-metrics-certs\") pod \"b7dda9a6-e738-403b-8b3a-2aa22356b419\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") "
Nov 03 08:23:06.765145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.765115    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7dda9a6-e738-403b-8b3a-2aa22356b419-kube-api-access-2qnkn\" (UniqueName: \"kubernetes.io/projected/b7dda9a6-e738-403b-8b3a-2aa22356b419-kube-api-access-2qnkn\") pod \"b7dda9a6-e738-403b-8b3a-2aa22356b419\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") "
Nov 03 08:23:06.765496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.765434    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-maas-api-tls\") pod \"b7dda9a6-e738-403b-8b3a-2aa22356b419\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") "
Nov 03 08:23:06.765687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.765652    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7dda9a6-e738-403b-8b3a-2aa22356b419-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dda9a6-e738-403b-8b3a-2aa22356b419-openshift-service-ca\") pod \"b7dda9a6-e738-403b-8b3a-2aa22356b419\" (UID: \"b7dda9a6-e738-403b-8b3a-2aa22356b419\") "
Nov 03 08:23:06.783712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.783617    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7dda9a6-e738-403b-8b3a-2aa22356b419-openshift-service-ca" pod "b7dda9a6-e738-403b-8b3a-2aa22356b419" (UID: "b7dda9a6-e738-403b-8b3a-2aa22356b419"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:06.826426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.826371    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-maas-api-tls" pod "b7dda9a6-e738-403b-8b3a-2aa22356b419" (UID: "b7dda9a6-e738-403b-8b3a-2aa22356b419"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:06.832553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.831364    6430 generic.go:356] "Generic (PLEG): container finished" podID="b7dda9a6-e738-403b-8b3a-2aa22356b419" containerID="5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f" exitCode=0
Nov 03 08:23:06.832553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.831570    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl" event={"ID":"b7dda9a6-e738-403b-8b3a-2aa22356b419","Type":"ContainerDied","Data":"5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f"}
Nov 03 08:23:06.832553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.831626    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl" event={"ID":"b7dda9a6-e738-403b-8b3a-2aa22356b419","Type":"ContainerDied","Data":"4bdc795b8722709f2786957dcc51aca1b066d399c78d390ca0ceeac8bedd6cb4"}
Nov 03 08:23:06.832553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.831657    6430 scope.go:122] "RemoveContainer" containerID="5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f"
Nov 03 08:23:06.832553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.831801    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"
Nov 03 08:23:06.854160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.854083    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-metrics-certs" pod "b7dda9a6-e738-403b-8b3a-2aa22356b419" (UID: "b7dda9a6-e738-403b-8b3a-2aa22356b419"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:06.933089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.929618    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7dda9a6-e738-403b-8b3a-2aa22356b419-kube-api-access-2qnkn" pod "b7dda9a6-e738-403b-8b3a-2aa22356b419" (UID: "b7dda9a6-e738-403b-8b3a-2aa22356b419"). InnerVolumeSpecName "kube-api-access-2qnkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:06.962273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.962222    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:06.964729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.964673    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.964958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.964938    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qnkn\" (UniqueName: \"kubernetes.io/projected/b7dda9a6-e738-403b-8b3a-2aa22356b419-kube-api-access-2qnkn\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.965109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.965093    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dda9a6-e738-403b-8b3a-2aa22356b419-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.965276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.965260    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dda9a6-e738-403b-8b3a-2aa22356b419-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.995987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.995964    6430 scope.go:122] "RemoveContainer" containerID="5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f"
Nov 03 08:23:06.996371 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:06.996350    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f\": container with ID starting with 5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f not found: ID does not exist" containerID="5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f"
Nov 03 08:23:06.996432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:06.996383    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f"} err="failed to get container status \"5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f\": rpc error: code = NotFound desc = could not find container \"5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f\": container with ID starting with 5a234f13a8da2c361cdd8196bc28430f7a5e3752357c600339b9199ad20fbd0f not found: ID does not exist"
Nov 03 08:23:07.085263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:07.085195    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:07.223360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:07.223329    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"]
Nov 03 08:23:07.227395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:07.227367    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-179c16-56db974668-kmkjl"]
Nov 03 08:23:07.418440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:07.418361    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:08.065972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:08.065936    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:08.620361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:08.620331    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7dda9a6-e738-403b-8b3a-2aa22356b419" path="/var/lib/kubelet/pods/b7dda9a6-e738-403b-8b3a-2aa22356b419/volumes"
Nov 03 08:23:08.881703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:08.881628    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:09.759820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:09.759780    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:10.063956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:10.063867    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" podUID="475aa1d3-6841-4350-959b-85d2ac5517bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:10.708583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:10.688581    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:11.588790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:11.588751    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:11.974598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:11.974536    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:12.101480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:12.101446    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:12.264810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:12.264738    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:12.404089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:12.401258    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:12.778085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:12.741356    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:12.778085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:12.753999    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"]
Nov 03 08:23:12.778085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:12.755440    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw" podUID="9fe323dc-a106-4bea-a996-d0602f6c7323" containerName="maas-api" containerID="cri-o://81b490298990097262a1891ef8043ec01942ece72b910904bcc213adef15e6b5" gracePeriod=30
Nov 03 08:23:12.872413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:12.872295    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:13.084497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.084368    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:13.144088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.126735    6430 generic.go:356] "Generic (PLEG): container finished" podID="9fe323dc-a106-4bea-a996-d0602f6c7323" containerID="81b490298990097262a1891ef8043ec01942ece72b910904bcc213adef15e6b5" exitCode=0
Nov 03 08:23:13.144088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.127185    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw" event={"ID":"9fe323dc-a106-4bea-a996-d0602f6c7323","Type":"ContainerDied","Data":"81b490298990097262a1891ef8043ec01942ece72b910904bcc213adef15e6b5"}
Nov 03 08:23:13.228583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.228473    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:13.614830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.614794    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:13.703086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.701017    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:23:13.783119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.782992    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:13.889980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.889937    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-maas-api-tls\") pod \"9fe323dc-a106-4bea-a996-d0602f6c7323\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") "
Nov 03 08:23:13.890205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.890002    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-metrics-certs\") pod \"9fe323dc-a106-4bea-a996-d0602f6c7323\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") "
Nov 03 08:23:13.890275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.890223    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe323dc-a106-4bea-a996-d0602f6c7323-kube-api-access-czv4n\" (UniqueName: \"kubernetes.io/projected/9fe323dc-a106-4bea-a996-d0602f6c7323-kube-api-access-czv4n\") pod \"9fe323dc-a106-4bea-a996-d0602f6c7323\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") "
Nov 03 08:23:13.890275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.890258    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe323dc-a106-4bea-a996-d0602f6c7323-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe323dc-a106-4bea-a996-d0602f6c7323-openshift-service-ca\") pod \"9fe323dc-a106-4bea-a996-d0602f6c7323\" (UID: \"9fe323dc-a106-4bea-a996-d0602f6c7323\") "
Nov 03 08:23:13.890614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.890591    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe323dc-a106-4bea-a996-d0602f6c7323-openshift-service-ca" pod "9fe323dc-a106-4bea-a996-d0602f6c7323" (UID: "9fe323dc-a106-4bea-a996-d0602f6c7323"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:13.891336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.891308    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe323dc-a106-4bea-a996-d0602f6c7323-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:13.892534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.892512    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-metrics-certs" pod "9fe323dc-a106-4bea-a996-d0602f6c7323" (UID: "9fe323dc-a106-4bea-a996-d0602f6c7323"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:13.892629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.892572    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-maas-api-tls" pod "9fe323dc-a106-4bea-a996-d0602f6c7323" (UID: "9fe323dc-a106-4bea-a996-d0602f6c7323"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:13.892690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.892636    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe323dc-a106-4bea-a996-d0602f6c7323-kube-api-access-czv4n" pod "9fe323dc-a106-4bea-a996-d0602f6c7323" (UID: "9fe323dc-a106-4bea-a996-d0602f6c7323"). InnerVolumeSpecName "kube-api-access-czv4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:13.992774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.992739    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:13.992774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.992770    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czv4n\" (UniqueName: \"kubernetes.io/projected/9fe323dc-a106-4bea-a996-d0602f6c7323-kube-api-access-czv4n\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:13.992774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:13.992780    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe323dc-a106-4bea-a996-d0602f6c7323-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:14.013215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.013176    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:14.136782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.136703    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"
Nov 03 08:23:14.136782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.136727    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw" event={"ID":"9fe323dc-a106-4bea-a996-d0602f6c7323","Type":"ContainerDied","Data":"0ddcdf00636454653790bcbffccfd5395180936aa4772ced784cd42e5be5aced"}
Nov 03 08:23:14.137022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.136786    6430 scope.go:122] "RemoveContainer" containerID="81b490298990097262a1891ef8043ec01942ece72b910904bcc213adef15e6b5"
Nov 03 08:23:14.162850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.162812    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"]
Nov 03 08:23:14.166628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.166597    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-fe584a-6c87bf9586-vcnlw"]
Nov 03 08:23:14.253369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.246615    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:14.385972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.385942    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:14.515982 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.515945    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:14.737117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.724452    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe323dc-a106-4bea-a996-d0602f6c7323" path="/var/lib/kubelet/pods/9fe323dc-a106-4bea-a996-d0602f6c7323/volumes"
Nov 03 08:23:14.894277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:14.893899    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:15.085774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.085701    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:23:15.150702 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.150667    6430 generic.go:356] "Generic (PLEG): container finished" podID="475aa1d3-6841-4350-959b-85d2ac5517bf" containerID="843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd" exitCode=0
Nov 03 08:23:15.150868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.150733    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"
Nov 03 08:23:15.150868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.150760    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" event={"ID":"475aa1d3-6841-4350-959b-85d2ac5517bf","Type":"ContainerDied","Data":"843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd"}
Nov 03 08:23:15.150868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.150832    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6" event={"ID":"475aa1d3-6841-4350-959b-85d2ac5517bf","Type":"ContainerDied","Data":"b78474a056b4ba7a328441a3337e5d96ab50864df8c4154875de45b7174ff084"}
Nov 03 08:23:15.150868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.150864    6430 scope.go:122] "RemoveContainer" containerID="843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd"
Nov 03 08:23:15.163981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.163960    6430 scope.go:122] "RemoveContainer" containerID="843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd"
Nov 03 08:23:15.164301 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:15.164277    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd\": container with ID starting with 843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd not found: ID does not exist" containerID="843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd"
Nov 03 08:23:15.164404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.164312    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd"} err="failed to get container status \"843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd\": rpc error: code = NotFound desc = could not find container \"843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd\": container with ID starting with 843b7703936147888a5560eaf74c1aa5652939e34d33ec2f0a728a5b6c6968cd not found: ID does not exist"
Nov 03 08:23:15.257888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.257856    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475aa1d3-6841-4350-959b-85d2ac5517bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475aa1d3-6841-4350-959b-85d2ac5517bf-plugins-config-volume\") pod \"475aa1d3-6841-4350-959b-85d2ac5517bf\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") "
Nov 03 08:23:15.258091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.257985    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475aa1d3-6841-4350-959b-85d2ac5517bf-kube-api-access-rdv9d\" (UniqueName: \"kubernetes.io/projected/475aa1d3-6841-4350-959b-85d2ac5517bf-kube-api-access-rdv9d\") pod \"475aa1d3-6841-4350-959b-85d2ac5517bf\" (UID: \"475aa1d3-6841-4350-959b-85d2ac5517bf\") "
Nov 03 08:23:15.258287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.258263    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475aa1d3-6841-4350-959b-85d2ac5517bf-plugins-config-volume" pod "475aa1d3-6841-4350-959b-85d2ac5517bf" (UID: "475aa1d3-6841-4350-959b-85d2ac5517bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:15.258794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.258772    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475aa1d3-6841-4350-959b-85d2ac5517bf-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:15.260339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.260311    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475aa1d3-6841-4350-959b-85d2ac5517bf-kube-api-access-rdv9d" pod "475aa1d3-6841-4350-959b-85d2ac5517bf" (UID: "475aa1d3-6841-4350-959b-85d2ac5517bf"). InnerVolumeSpecName "kube-api-access-rdv9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:15.359936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.359850    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdv9d\" (UniqueName: \"kubernetes.io/projected/475aa1d3-6841-4350-959b-85d2ac5517bf-kube-api-access-rdv9d\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:15.474717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.474685    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"]
Nov 03 08:23:15.478502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.478475    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-179c16-7ff7885767-pbzc6"]
Nov 03 08:23:15.753293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.753261    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:15.847886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.847845    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"]
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848797    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" containerName="payload-pre-processing"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848818    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" containerName="payload-pre-processing"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848836    6430 cpu_manager.go:414] "containerMap: removing container" podUID="475aa1d3-6841-4350-959b-85d2ac5517bf" containerName="payload-pre-processing"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848841    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475aa1d3-6841-4350-959b-85d2ac5517bf" containerName="payload-pre-processing"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848854    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9ec8f62e-72ce-485b-868e-c142e9d14291" containerName="maas-api"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848861    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec8f62e-72ce-485b-868e-c142e9d14291" containerName="maas-api"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848884    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9fe323dc-a106-4bea-a996-d0602f6c7323" containerName="maas-api"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848889    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe323dc-a106-4bea-a996-d0602f6c7323" containerName="maas-api"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848909    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b7dda9a6-e738-403b-8b3a-2aa22356b419" containerName="maas-api"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848914    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7dda9a6-e738-403b-8b3a-2aa22356b419" containerName="maas-api"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848935    6430 cpu_manager.go:414] "containerMap: removing container" podUID="be030d14-64df-4b28-8224-145b22b369a5" containerName="payload-processing"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848940    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be030d14-64df-4b28-8224-145b22b369a5" containerName="payload-processing"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848955    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fcadc141-2f8d-4474-81cd-9aeac02c02aa" containerName="maas-api"
Nov 03 08:23:15.849100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.848963    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcadc141-2f8d-4474-81cd-9aeac02c02aa" containerName="maas-api"
Nov 03 08:23:15.849926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.849138    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="be030d14-64df-4b28-8224-145b22b369a5" containerName="payload-processing"
Nov 03 08:23:15.849926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.849153    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcadc141-2f8d-4474-81cd-9aeac02c02aa" containerName="maas-api"
Nov 03 08:23:15.849926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.849170    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9ad8b7-51ea-4317-9ec1-2d9b9cb0ac9b" containerName="payload-pre-processing"
Nov 03 08:23:15.849926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.849181    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="475aa1d3-6841-4350-959b-85d2ac5517bf" containerName="payload-pre-processing"
Nov 03 08:23:15.849926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.849188    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec8f62e-72ce-485b-868e-c142e9d14291" containerName="maas-api"
Nov 03 08:23:15.849926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.849197    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe323dc-a106-4bea-a996-d0602f6c7323" containerName="maas-api"
Nov 03 08:23:15.849926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.849208    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7dda9a6-e738-403b-8b3a-2aa22356b419" containerName="maas-api"
Nov 03 08:23:15.855186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.854591    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.858027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.857788    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a61a48\"/\"default-dockercfg-v79tr\""
Nov 03 08:23:15.863995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.860169    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a61a48\"/\"openshift-service-ca.crt\""
Nov 03 08:23:15.863995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.860245    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a61a48\"/\"auth-test-model-a61a48-kserve-self-signed-certs\""
Nov 03 08:23:15.863995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.860274    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a61a48\"/\"kube-root-ca.crt\""
Nov 03 08:23:15.867214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.867183    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-dshm\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.867393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.867299    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-model-cache\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.867393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.867333    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-home\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.867573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.867475    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxq4v\" (UniqueName: \"kubernetes.io/projected/2387acc4-0876-413d-9714-aafb39f01ee6-kube-api-access-hxq4v\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.868283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.868027    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-tmp-dir\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.868283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.868143    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2387acc4-0876-413d-9714-aafb39f01ee6-tls-certs\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.882691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.882407    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"]
Nov 03 08:23:15.910200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.910169    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:15.970011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.969985    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-model-cache\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970017    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-home\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970174    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxq4v\" (UniqueName: \"kubernetes.io/projected/2387acc4-0876-413d-9714-aafb39f01ee6-kube-api-access-hxq4v\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970306    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-tmp-dir\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970467 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970449    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-model-cache\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970493    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-home\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970587    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-tmp-dir\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970606    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2387acc4-0876-413d-9714-aafb39f01ee6-tls-certs\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.970767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.970663    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-dshm\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.972944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.972918    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-dshm\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.973245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.973223    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2387acc4-0876-413d-9714-aafb39f01ee6-tls-certs\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:15.978459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:15.978432    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxq4v\" (UniqueName: \"kubernetes.io/projected/2387acc4-0876-413d-9714-aafb39f01ee6-kube-api-access-hxq4v\") pod \"auth-test-model-a61a48-kserve-856c756b8f-p6bng\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") " pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:16.208087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:16.184685    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:16.582242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:16.582207    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"]
Nov 03 08:23:16.583645 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:23:16.583616    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2387acc4_0876_413d_9714_aafb39f01ee6.slice/crio-3bfd3d959e39a83b10572419d0a4a68a8901a078f57a87c7a8039f0435070a27 WatchSource:0}: Error finding container 3bfd3d959e39a83b10572419d0a4a68a8901a078f57a87c7a8039f0435070a27: Status 404 returned error can't find the container with id 3bfd3d959e39a83b10572419d0a4a68a8901a078f57a87c7a8039f0435070a27
Nov 03 08:23:16.621281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:16.621249    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475aa1d3-6841-4350-959b-85d2ac5517bf" path="/var/lib/kubelet/pods/475aa1d3-6841-4350-959b-85d2ac5517bf/volumes"
Nov 03 08:23:16.742162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:16.742128    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:17.268575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:17.268531    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng" event={"ID":"2387acc4-0876-413d-9714-aafb39f01ee6","Type":"ContainerStarted","Data":"3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b"}
Nov 03 08:23:17.268575 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:17.268575    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng" event={"ID":"2387acc4-0876-413d-9714-aafb39f01ee6","Type":"ContainerStarted","Data":"3bfd3d959e39a83b10572419d0a4a68a8901a078f57a87c7a8039f0435070a27"}
Nov 03 08:23:17.284994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:17.284944    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng" podStartSLOduration=2.284931651 podStartE2EDuration="2.284931651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:17.282356922 +0000 UTC m=+1746.785192435" watchObservedRunningTime="2026-11-03 08:23:17.284931651 +0000 UTC m=+1746.787767164"
Nov 03 08:23:17.634157 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:17.634032    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:18.180804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:18.180752    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:18.192465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:18.192434    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:18.452510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:18.452477    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:19.315531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:19.315497    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:20.081492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:20.081449    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:20.279259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:20.279225    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:20.413909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:20.413809    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"]
Nov 03 08:23:20.414277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:20.414076    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy" containerID="cri-o://ac5d2f23d2e52733f714494c9c02dc13eea515b49ae6e525b9a82b1d4d0aa8f0" gracePeriod=30
Nov 03 08:23:20.444638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:20.444604    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:20.820558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:20.820445    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:20.939532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:20.939494    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:21.535962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:21.535932    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:21.971580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:21.971528    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"]
Nov 03 08:23:22.025641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:22.025606    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:22.297164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:22.290403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:22.421017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:22.420972    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:22.926764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:22.926731    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:23.698426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:23.698392    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:23.813045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:23.813008    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:24.007637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:24.007555    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"]
Nov 03 08:23:24.072416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:24.072377    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:24.757786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:24.757742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:25.529639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:25.529607    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:25.831890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:25.831855    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"]
Nov 03 08:23:25.952237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:25.952181    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:26.192123 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.192051    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:26.205122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.205086    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:26.436446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.436381    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:26.561784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.561757    6430 generic.go:356] "Generic (PLEG): container finished" podID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerID="ac5d2f23d2e52733f714494c9c02dc13eea515b49ae6e525b9a82b1d4d0aa8f0" exitCode=0
Nov 03 08:23:26.562391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.561946    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" event={"ID":"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda","Type":"ContainerDied","Data":"ac5d2f23d2e52733f714494c9c02dc13eea515b49ae6e525b9a82b1d4d0aa8f0"}
Nov 03 08:23:26.562599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.562583    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb" event={"ID":"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda","Type":"ContainerDied","Data":"931d3083aab0b8c9cc56c4acd038d47ae7a5f17068f403ac7dee83d361b1fac4"}
Nov 03 08:23:26.562707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.562695    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931d3083aab0b8c9cc56c4acd038d47ae7a5f17068f403ac7dee83d361b1fac4"
Nov 03 08:23:26.562791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.562783    6430 scope.go:122] "RemoveContainer" containerID="14bfff959d007fa00cca4446875837aab2b419670c3e949a9b20f5983f20f76f"
Nov 03 08:23:26.566684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.566363    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:23:26.577559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.577538    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:23:26.671567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671524    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-envoy\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.671567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671577    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-podinfo\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.671833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671646    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-certs\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.671833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671738    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-token\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-token\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.671938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671859    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-credential-socket\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.671938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671877    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istiod-ca-cert\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.671938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671911    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-socket\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.671938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.671933    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-kube-api-access-zgg5s\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-kube-api-access-zgg5s\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.672164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.672001    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-data\") pod \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\" (UID: \"41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda\") "
Nov 03 08:23:26.672656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.672624    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-certs" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:26.672656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.672646    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-socket" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:26.672820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.672707    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istiod-ca-cert" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:26.672820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.672782    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.672820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.672804    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.672972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.672834    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-data" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:26.673249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.673216    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-credential-socket" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:26.675294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.675268    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-kube-api-access-zgg5s" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "kube-api-access-zgg5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:26.675421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.675391    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-token" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:26.676594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.676569    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-envoy" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:26.678042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.678012    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-podinfo" pod "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" (UID: "41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:23:26.775422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.775343    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.775422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.775382    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.775422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.775399    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.775422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.775408    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgg5s\" (UniqueName: \"kubernetes.io/projected/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-kube-api-access-zgg5s\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.775422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.775418    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.775422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.775427    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.775741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:26.775435    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:27.237653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:27.237620    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:27.571231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:27.571158    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"
Nov 03 08:23:27.605686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:27.605658    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"]
Nov 03 08:23:27.610806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:27.610778    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-fe584a-openshift-default-7754955887-d6cwb"]
Nov 03 08:23:28.042411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:28.042373    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:28.380095 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:28.379951    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:23:28.619782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:28.619752    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" path="/var/lib/kubelet/pods/41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda/volumes"
Nov 03 08:23:28.799500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:28.799462    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:29.767393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:29.767360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:30.458825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:30.458789    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:31.320883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:31.320831    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:31.536105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:31.536033    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:31.755677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:31.755626    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:32.474042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:32.474004    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:33.280776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:33.280736    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:33.416188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:33.416151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:34.158042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.158003    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"]
Nov 03 08:23:34.159747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.159716    6430 cpu_manager.go:414] "containerMap: removing container" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy"
Nov 03 08:23:34.159747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.159749    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy"
Nov 03 08:23:34.160003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.159987    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy"
Nov 03 08:23:34.160107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.160018    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy"
Nov 03 08:23:34.165013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.164990    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.171033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.171005    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"]
Nov 03 08:23:34.171560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.171527    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6a8bf\"/\"openshift-service-ca.crt\""
Nov 03 08:23:34.171686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.171527    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6a8bf\"/\"auth-test-model-d6a8bf-kserve-self-signed-certs\""
Nov 03 08:23:34.171686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.171577    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6a8bf\"/\"default-dockercfg-zdw9p\""
Nov 03 08:23:34.171933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.171915    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d6a8bf\"/\"kube-root-ca.crt\""
Nov 03 08:23:34.220634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.220607    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:34.260335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.260306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-tmp-dir\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.260503 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.260458    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-home\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.260613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.260596    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04905656-0d4f-427d-a89f-9eba8e9a17b0-tls-certs\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.260672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.260658    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-dshm\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.260806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.260788    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-model-cache\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.260876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.260816    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s52d\" (UniqueName: \"kubernetes.io/projected/04905656-0d4f-427d-a89f-9eba8e9a17b0-kube-api-access-9s52d\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.361970    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-dshm\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362088    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-model-cache\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362108    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s52d\" (UniqueName: \"kubernetes.io/projected/04905656-0d4f-427d-a89f-9eba8e9a17b0-kube-api-access-9s52d\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362180    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-tmp-dir\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362270    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-home\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362417    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04905656-0d4f-427d-a89f-9eba8e9a17b0-tls-certs\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362560    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-model-cache\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362678    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-tmp-dir\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.362897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.362875    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-home\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.364700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.364632    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:34.365021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.364993    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-dshm\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.365409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.365385    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04905656-0d4f-427d-a89f-9eba8e9a17b0-tls-certs\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.373935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.373907    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s52d\" (UniqueName: \"kubernetes.io/projected/04905656-0d4f-427d-a89f-9eba8e9a17b0-kube-api-access-9s52d\") pod \"auth-test-model-d6a8bf-kserve-b8c976889-4bd6f\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") " pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.492199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.491881    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:34.936886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:34.936856    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"]
Nov 03 08:23:34.940180 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:23:34.940147    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04905656_0d4f_427d_a89f_9eba8e9a17b0.slice/crio-4e0e0f42b871a82ebc2c52541fdf21532f824afc0f2464b06fe5eb63b438b5ee WatchSource:0}: Error finding container 4e0e0f42b871a82ebc2c52541fdf21532f824afc0f2464b06fe5eb63b438b5ee: Status 404 returned error can't find the container with id 4e0e0f42b871a82ebc2c52541fdf21532f824afc0f2464b06fe5eb63b438b5ee
Nov 03 08:23:35.330897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:35.330768    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:35.722706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:35.722674    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f" event={"ID":"04905656-0d4f-427d-a89f-9eba8e9a17b0","Type":"ContainerStarted","Data":"8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775"}
Nov 03 08:23:35.722902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:35.722717    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f" event={"ID":"04905656-0d4f-427d-a89f-9eba8e9a17b0","Type":"ContainerStarted","Data":"4e0e0f42b871a82ebc2c52541fdf21532f824afc0f2464b06fe5eb63b438b5ee"}
Nov 03 08:23:35.742442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:35.742393    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f" podStartSLOduration=1.742379463 podStartE2EDuration="1.742379463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:35.737439408 +0000 UTC m=+1765.240274923" watchObservedRunningTime="2026-11-03 08:23:35.742379463 +0000 UTC m=+1765.245215011"
Nov 03 08:23:36.129150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:36.129052    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:36.493531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:36.493494    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:36.988093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:36.987600    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:37.008888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.008845    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q" podUID="a9cf911f-3512-4745-a08b-bf6616df4894" containerName="main" containerID="cri-o://e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd" gracePeriod=45
Nov 03 08:23:37.035707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.035675    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"]
Nov 03 08:23:37.145210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.127844    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:37.561974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.561952    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:23:37.677289 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677216    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-home\") pod \"a9cf911f-3512-4745-a08b-bf6616df4894\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") "
Nov 03 08:23:37.677289 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677267    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cf911f-3512-4745-a08b-bf6616df4894-kube-api-access-7r5cz\" (UniqueName: \"kubernetes.io/projected/a9cf911f-3512-4745-a08b-bf6616df4894-kube-api-access-7r5cz\") pod \"a9cf911f-3512-4745-a08b-bf6616df4894\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") "
Nov 03 08:23:37.677471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677328    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-model-cache\") pod \"a9cf911f-3512-4745-a08b-bf6616df4894\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") "
Nov 03 08:23:37.677471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677369    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-dshm\") pod \"a9cf911f-3512-4745-a08b-bf6616df4894\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") "
Nov 03 08:23:37.677570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677544    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-home" pod "a9cf911f-3512-4745-a08b-bf6616df4894" (UID: "a9cf911f-3512-4745-a08b-bf6616df4894"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:37.677614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677553    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-tmp-dir\") pod \"a9cf911f-3512-4745-a08b-bf6616df4894\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") "
Nov 03 08:23:37.677614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677571    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-model-cache" pod "a9cf911f-3512-4745-a08b-bf6616df4894" (UID: "a9cf911f-3512-4745-a08b-bf6616df4894"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:37.677719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677625    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9cf911f-3512-4745-a08b-bf6616df4894-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf911f-3512-4745-a08b-bf6616df4894-tls-certs\") pod \"a9cf911f-3512-4745-a08b-bf6616df4894\" (UID: \"a9cf911f-3512-4745-a08b-bf6616df4894\") "
Nov 03 08:23:37.677783 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.677738    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-tmp-dir" pod "a9cf911f-3512-4745-a08b-bf6616df4894" (UID: "a9cf911f-3512-4745-a08b-bf6616df4894"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:37.679285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.679263    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:37.679285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.679286    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:37.679494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.679299    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:37.680180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.680161    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-dshm" pod "a9cf911f-3512-4745-a08b-bf6616df4894" (UID: "a9cf911f-3512-4745-a08b-bf6616df4894"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:37.680282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.680179    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cf911f-3512-4745-a08b-bf6616df4894-tls-certs" pod "a9cf911f-3512-4745-a08b-bf6616df4894" (UID: "a9cf911f-3512-4745-a08b-bf6616df4894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:37.680282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.680214    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cf911f-3512-4745-a08b-bf6616df4894-kube-api-access-7r5cz" pod "a9cf911f-3512-4745-a08b-bf6616df4894" (UID: "a9cf911f-3512-4745-a08b-bf6616df4894"). InnerVolumeSpecName "kube-api-access-7r5cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:37.739779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.739745    6430 generic.go:356] "Generic (PLEG): container finished" podID="a9cf911f-3512-4745-a08b-bf6616df4894" containerID="e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd" exitCode=0
Nov 03 08:23:37.739922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.739787    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"
Nov 03 08:23:37.739922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.739826    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q" event={"ID":"a9cf911f-3512-4745-a08b-bf6616df4894","Type":"ContainerDied","Data":"e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd"}
Nov 03 08:23:37.739922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.739871    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q" event={"ID":"a9cf911f-3512-4745-a08b-bf6616df4894","Type":"ContainerDied","Data":"8dff78e367fd548d4a11bdfb932e58aa3708b30197443cfdaabb9281cac6fe77"}
Nov 03 08:23:37.739922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.739892    6430 scope.go:122] "RemoveContainer" containerID="e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd"
Nov 03 08:23:37.757969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.757928    6430 scope.go:122] "RemoveContainer" containerID="e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd"
Nov 03 08:23:37.758491 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:37.758373    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd\": container with ID starting with e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd not found: ID does not exist" containerID="e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd"
Nov 03 08:23:37.758491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.758410    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd"} err="failed to get container status \"e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd\": rpc error: code = NotFound desc = could not find container \"e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd\": container with ID starting with e8ce45a968cbba5a2e54ca2458c1ba213c59ac8cc83535a6ceb5ee635bf9cadd not found: ID does not exist"
Nov 03 08:23:37.762993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.762966    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"]
Nov 03 08:23:37.766996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.766972    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e0f576/test-model-e0f576-kserve-78fcf7b565-97s6q"]
Nov 03 08:23:37.780361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.780334    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5cz\" (UniqueName: \"kubernetes.io/projected/a9cf911f-3512-4745-a08b-bf6616df4894-kube-api-access-7r5cz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:37.780361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.780358    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf911f-3512-4745-a08b-bf6616df4894-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:37.780496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.780369    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf911f-3512-4745-a08b-bf6616df4894-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:37.952616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:37.952583    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:38.220532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:38.220346    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"]
Nov 03 08:23:38.266440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:38.266403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:38.633692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:38.633605    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cf911f-3512-4745-a08b-bf6616df4894" path="/var/lib/kubelet/pods/a9cf911f-3512-4745-a08b-bf6616df4894/volumes"
Nov 03 08:23:39.060348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.060310    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" podUID="607fca89-8aa0-4714-820a-393ab37084c0" containerName="main" containerID="cri-o://03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666" gracePeriod=45
Nov 03 08:23:39.297087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.297034    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:39.510306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.510267    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:23:39.668134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668103    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607fca89-8aa0-4714-820a-393ab37084c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/607fca89-8aa0-4714-820a-393ab37084c0-tls-certs\") pod \"607fca89-8aa0-4714-820a-393ab37084c0\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") "
Nov 03 08:23:39.668134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668137    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-home\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-home\") pod \"607fca89-8aa0-4714-820a-393ab37084c0\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") "
Nov 03 08:23:39.668691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668240    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-model-cache\") pod \"607fca89-8aa0-4714-820a-393ab37084c0\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") "
Nov 03 08:23:39.668691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668389    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-tmp-dir\") pod \"607fca89-8aa0-4714-820a-393ab37084c0\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") "
Nov 03 08:23:39.668691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668462    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-home" pod "607fca89-8aa0-4714-820a-393ab37084c0" (UID: "607fca89-8aa0-4714-820a-393ab37084c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:39.668691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668504    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607fca89-8aa0-4714-820a-393ab37084c0-kube-api-access-2klgd\" (UniqueName: \"kubernetes.io/projected/607fca89-8aa0-4714-820a-393ab37084c0-kube-api-access-2klgd\") pod \"607fca89-8aa0-4714-820a-393ab37084c0\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") "
Nov 03 08:23:39.668691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668593    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-dshm\") pod \"607fca89-8aa0-4714-820a-393ab37084c0\" (UID: \"607fca89-8aa0-4714-820a-393ab37084c0\") "
Nov 03 08:23:39.668691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.668667    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-tmp-dir" pod "607fca89-8aa0-4714-820a-393ab37084c0" (UID: "607fca89-8aa0-4714-820a-393ab37084c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:39.669142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.669099    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-model-cache" pod "607fca89-8aa0-4714-820a-393ab37084c0" (UID: "607fca89-8aa0-4714-820a-393ab37084c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:39.669508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.669482    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:39.669608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.669509    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:39.669608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.669524    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:39.670707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.670681    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607fca89-8aa0-4714-820a-393ab37084c0-tls-certs" pod "607fca89-8aa0-4714-820a-393ab37084c0" (UID: "607fca89-8aa0-4714-820a-393ab37084c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:39.670924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.670908    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607fca89-8aa0-4714-820a-393ab37084c0-kube-api-access-2klgd" pod "607fca89-8aa0-4714-820a-393ab37084c0" (UID: "607fca89-8aa0-4714-820a-393ab37084c0"). InnerVolumeSpecName "kube-api-access-2klgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:39.670924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.670915    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-dshm" pod "607fca89-8aa0-4714-820a-393ab37084c0" (UID: "607fca89-8aa0-4714-820a-393ab37084c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:39.757945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.757860    6430 generic.go:356] "Generic (PLEG): container finished" podID="607fca89-8aa0-4714-820a-393ab37084c0" containerID="03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666" exitCode=0
Nov 03 08:23:39.757945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.757922    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"
Nov 03 08:23:39.758188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.757951    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" event={"ID":"607fca89-8aa0-4714-820a-393ab37084c0","Type":"ContainerDied","Data":"03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666"}
Nov 03 08:23:39.758188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.758001    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" event={"ID":"607fca89-8aa0-4714-820a-393ab37084c0","Type":"ContainerDied","Data":"d9d207537826f29431bdab7c76766b37e8e6ffe6d3658d01fde200adb47e2d7a"}
Nov 03 08:23:39.758188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.758022    6430 scope.go:122] "RemoveContainer" containerID="03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666"
Nov 03 08:23:39.770389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.770373    6430 scope.go:122] "RemoveContainer" containerID="03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666"
Nov 03 08:23:39.770785 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:39.770753    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666\": container with ID starting with 03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666 not found: ID does not exist" containerID="03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666"
Nov 03 08:23:39.770892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.770795    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666"} err="failed to get container status \"03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666\": rpc error: code = NotFound desc = could not find container \"03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666\": container with ID starting with 03c1171ac2cfc4bc8a671a631b382574e651c94e47844076ac13cffe1370e666 not found: ID does not exist"
Nov 03 08:23:39.771203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.771182    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2klgd\" (UniqueName: \"kubernetes.io/projected/607fca89-8aa0-4714-820a-393ab37084c0-kube-api-access-2klgd\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:39.771317 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.771209    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607fca89-8aa0-4714-820a-393ab37084c0-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:39.771317 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.771225    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607fca89-8aa0-4714-820a-393ab37084c0-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:39.780876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.780856    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"]
Nov 03 08:23:39.785344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:39.785325    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7"]
Nov 03 08:23:40.225291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:40.225256    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:40.346271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:40.346229    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9b559e/test-model-9b559e-kserve-7867d7dfbf-q62s7" podUID="607fca89-8aa0-4714-820a-393ab37084c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:23:40.621376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:40.621293    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607fca89-8aa0-4714-820a-393ab37084c0" path="/var/lib/kubelet/pods/607fca89-8aa0-4714-820a-393ab37084c0/volumes"
Nov 03 08:23:40.856482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:40.856441    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main" containerID="cri-o://9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a" gracePeriod=45
Nov 03 08:23:41.143087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.142992    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:23:41.184214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.184174    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:41.295784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.295744    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-tmp-dir\") pod \"35388dcd-6433-4f50-a084-947740876acb\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") "
Nov 03 08:23:41.295969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.295947    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-home\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-home\") pod \"35388dcd-6433-4f50-a084-947740876acb\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") "
Nov 03 08:23:41.296083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.296038    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-tmp-dir" pod "35388dcd-6433-4f50-a084-947740876acb" (UID: "35388dcd-6433-4f50-a084-947740876acb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:41.296158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.296090    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35388dcd-6433-4f50-a084-947740876acb-kube-api-access-8rxs5\" (UniqueName: \"kubernetes.io/projected/35388dcd-6433-4f50-a084-947740876acb-kube-api-access-8rxs5\") pod \"35388dcd-6433-4f50-a084-947740876acb\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") "
Nov 03 08:23:41.296158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.296128    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35388dcd-6433-4f50-a084-947740876acb-tls-certs\" (UniqueName: \"kubernetes.io/secret/35388dcd-6433-4f50-a084-947740876acb-tls-certs\") pod \"35388dcd-6433-4f50-a084-947740876acb\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") "
Nov 03 08:23:41.296278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.296255    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-home" pod "35388dcd-6433-4f50-a084-947740876acb" (UID: "35388dcd-6433-4f50-a084-947740876acb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:41.296401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.296382    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-dshm\") pod \"35388dcd-6433-4f50-a084-947740876acb\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") "
Nov 03 08:23:41.296468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.296418    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-model-cache\") pod \"35388dcd-6433-4f50-a084-947740876acb\" (UID: \"35388dcd-6433-4f50-a084-947740876acb\") "
Nov 03 08:23:41.297210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.297183    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-model-cache" pod "35388dcd-6433-4f50-a084-947740876acb" (UID: "35388dcd-6433-4f50-a084-947740876acb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:41.298638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.298611    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.298749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.298643    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.298749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.298658    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.307311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.307282    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35388dcd-6433-4f50-a084-947740876acb-kube-api-access-8rxs5" pod "35388dcd-6433-4f50-a084-947740876acb" (UID: "35388dcd-6433-4f50-a084-947740876acb"). InnerVolumeSpecName "kube-api-access-8rxs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:41.308780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.308752    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35388dcd-6433-4f50-a084-947740876acb-tls-certs" pod "35388dcd-6433-4f50-a084-947740876acb" (UID: "35388dcd-6433-4f50-a084-947740876acb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:41.309283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.309256    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-dshm" pod "35388dcd-6433-4f50-a084-947740876acb" (UID: "35388dcd-6433-4f50-a084-947740876acb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:41.399979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.399883    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxs5\" (UniqueName: \"kubernetes.io/projected/35388dcd-6433-4f50-a084-947740876acb-kube-api-access-8rxs5\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.399979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.399928    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35388dcd-6433-4f50-a084-947740876acb-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.399979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.399938    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35388dcd-6433-4f50-a084-947740876acb-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.774513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.774476    6430 generic.go:356] "Generic (PLEG): container finished" podID="35388dcd-6433-4f50-a084-947740876acb" containerID="9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a" exitCode=0
Nov 03 08:23:41.774690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.774526    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"
Nov 03 08:23:41.774690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.774558    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" event={"ID":"35388dcd-6433-4f50-a084-947740876acb","Type":"ContainerDied","Data":"9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a"}
Nov 03 08:23:41.774690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.774603    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz" event={"ID":"35388dcd-6433-4f50-a084-947740876acb","Type":"ContainerDied","Data":"d5ea27405540c670c117bf8eba4df9271f124b2fb02fd4be783d3c6214c8327c"}
Nov 03 08:23:41.774690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.774622    6430 scope.go:122] "RemoveContainer" containerID="9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a"
Nov 03 08:23:41.785790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.785767    6430 scope.go:122] "RemoveContainer" containerID="9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a"
Nov 03 08:23:41.786097 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:41.786047    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a\": container with ID starting with 9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a not found: ID does not exist" containerID="9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a"
Nov 03 08:23:41.786173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.786092    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a"} err="failed to get container status \"9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a\": rpc error: code = NotFound desc = could not find container \"9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a\": container with ID starting with 9dfa04f87f62b71a1649e9fe25cb9bfb0e3b0872e4e2d8d312b205d1521b276a not found: ID does not exist"
Nov 03 08:23:41.801996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.801972    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"]
Nov 03 08:23:41.817873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:41.817847    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/simulator-w0-76ef13-kserve-9cc46dd9c-hxwrz"]
Nov 03 08:23:42.146145 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:42.140439    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:42.396610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:42.373670    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:42.566104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:42.565464    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:42.666140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:42.664971    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35388dcd-6433-4f50-a084-947740876acb" path="/var/lib/kubelet/pods/35388dcd-6433-4f50-a084-947740876acb/volumes"
Nov 03 08:23:43.365188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:43.357540    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:43.490088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:43.488307    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:43.629464 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:43.629373    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:44.493754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:44.493722    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:44.513667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:44.513639    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:44.528905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:44.528873    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:44.822071 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:44.821976    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:23:45.682107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:45.682040    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:45.951799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:45.951668    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:46.114976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:46.114936    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:46.968025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:46.967991    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:47.348690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:47.348607    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"]
Nov 03 08:23:47.461125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:47.461093    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:47.586523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:47.586487    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:48.345343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:48.345194    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:48.766699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:48.766664    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:48.886144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:48.886032    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:49.444794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:49.444736    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:50.249022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:50.248992    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:51.177588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:51.177552    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:51.952277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:51.952231    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:52.115470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.115436    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:52.187087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.180425    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" podUID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" containerName="main" containerID="cri-o://e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377" gracePeriod=45
Nov 03 08:23:52.250647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.250599    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:52.319156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.319122    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" podUID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.148:8000/ready\": dial tcp 10.133.0.148:8000: connect: connection refused"
Nov 03 08:23:52.349614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.349577    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9" podUID="89fd0430-b072-458b-91fa-8a1459789c95" containerName="payload-processing" containerID="cri-o://c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63" gracePeriod=25
Nov 03 08:23:52.498555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.498518    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:23:52.505840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.505806    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.581222    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tls-certs\") pod \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") "
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.581397    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-home\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-home\") pod \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") "
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.581481    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-dshm\") pod \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") "
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.581517    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tmp-dir\") pod \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") "
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.581635    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee9f9ca-9623-4300-9b04-6c3ba857841d-kube-api-access-hcjfv\" (UniqueName: \"kubernetes.io/projected/7ee9f9ca-9623-4300-9b04-6c3ba857841d-kube-api-access-hcjfv\") pod \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") "
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.581828    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-model-cache\") pod \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\" (UID: \"7ee9f9ca-9623-4300-9b04-6c3ba857841d\") "
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.585501    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-home" pod "7ee9f9ca-9623-4300-9b04-6c3ba857841d" (UID: "7ee9f9ca-9623-4300-9b04-6c3ba857841d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:52.586352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.585708    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tmp-dir" pod "7ee9f9ca-9623-4300-9b04-6c3ba857841d" (UID: "7ee9f9ca-9623-4300-9b04-6c3ba857841d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:52.589704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.589653    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tls-certs" pod "7ee9f9ca-9623-4300-9b04-6c3ba857841d" (UID: "7ee9f9ca-9623-4300-9b04-6c3ba857841d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:52.593218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.593196    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-dshm" pod "7ee9f9ca-9623-4300-9b04-6c3ba857841d" (UID: "7ee9f9ca-9623-4300-9b04-6c3ba857841d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:52.593363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.593242    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee9f9ca-9623-4300-9b04-6c3ba857841d-kube-api-access-hcjfv" pod "7ee9f9ca-9623-4300-9b04-6c3ba857841d" (UID: "7ee9f9ca-9623-4300-9b04-6c3ba857841d"). InnerVolumeSpecName "kube-api-access-hcjfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:52.593363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.593318    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-model-cache" pod "7ee9f9ca-9623-4300-9b04-6c3ba857841d" (UID: "7ee9f9ca-9623-4300-9b04-6c3ba857841d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:52.628167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.628106    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:52.648413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.648385    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:23:52.690457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.690417    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.690457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.690456    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.690457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.690472    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.691269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.690509    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.691269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.690526    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjfv\" (UniqueName: \"kubernetes.io/projected/7ee9f9ca-9623-4300-9b04-6c3ba857841d-kube-api-access-hcjfv\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.691269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.690542    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee9f9ca-9623-4300-9b04-6c3ba857841d-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.792256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.791235    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fd0430-b072-458b-91fa-8a1459789c95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fd0430-b072-458b-91fa-8a1459789c95-plugins-config-volume\") pod \"89fd0430-b072-458b-91fa-8a1459789c95\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") "
Nov 03 08:23:52.792256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.791315    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fd0430-b072-458b-91fa-8a1459789c95-kube-api-access-fgcq6\" (UniqueName: \"kubernetes.io/projected/89fd0430-b072-458b-91fa-8a1459789c95-kube-api-access-fgcq6\") pod \"89fd0430-b072-458b-91fa-8a1459789c95\" (UID: \"89fd0430-b072-458b-91fa-8a1459789c95\") "
Nov 03 08:23:52.794832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.793249    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fd0430-b072-458b-91fa-8a1459789c95-plugins-config-volume" pod "89fd0430-b072-458b-91fa-8a1459789c95" (UID: "89fd0430-b072-458b-91fa-8a1459789c95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:52.861088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.854086    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:52.861088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.855020    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fd0430-b072-458b-91fa-8a1459789c95-kube-api-access-fgcq6" pod "89fd0430-b072-458b-91fa-8a1459789c95" (UID: "89fd0430-b072-458b-91fa-8a1459789c95"). InnerVolumeSpecName "kube-api-access-fgcq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:52.893691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.893654    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fd0430-b072-458b-91fa-8a1459789c95-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.893691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.893690    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgcq6\" (UniqueName: \"kubernetes.io/projected/89fd0430-b072-458b-91fa-8a1459789c95-kube-api-access-fgcq6\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:52.959641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.959594    6430 generic.go:356] "Generic (PLEG): container finished" podID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" containerID="e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377" exitCode=0
Nov 03 08:23:52.959641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.959642    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"
Nov 03 08:23:52.959902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.959662    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" event={"ID":"7ee9f9ca-9623-4300-9b04-6c3ba857841d","Type":"ContainerDied","Data":"e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377"}
Nov 03 08:23:52.959902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.959709    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb" event={"ID":"7ee9f9ca-9623-4300-9b04-6c3ba857841d","Type":"ContainerDied","Data":"c4074d727442b2bc2c0be05991d99351ddfb9b1638b3e4c0073441ddd05f3004"}
Nov 03 08:23:52.959902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.959731    6430 scope.go:122] "RemoveContainer" containerID="e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377"
Nov 03 08:23:52.961605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.961582    6430 generic.go:356] "Generic (PLEG): container finished" podID="89fd0430-b072-458b-91fa-8a1459789c95" containerID="c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63" exitCode=0
Nov 03 08:23:52.961797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.961636    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"
Nov 03 08:23:52.961889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.961810    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9" event={"ID":"89fd0430-b072-458b-91fa-8a1459789c95","Type":"ContainerDied","Data":"c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63"}
Nov 03 08:23:52.961889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.961860    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9" event={"ID":"89fd0430-b072-458b-91fa-8a1459789c95","Type":"ContainerDied","Data":"db528f663d22bf9f97a943121a23d4994b70aa4d12979727b4d79c6561885e6c"}
Nov 03 08:23:52.971224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.971203    6430 scope.go:122] "RemoveContainer" containerID="e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377"
Nov 03 08:23:52.971599 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:52.971579    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377\": container with ID starting with e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377 not found: ID does not exist" containerID="e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377"
Nov 03 08:23:52.971682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.971608    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377"} err="failed to get container status \"e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377\": rpc error: code = NotFound desc = could not find container \"e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377\": container with ID starting with e2938efc034d480993e6d085b468a92f4a7eaee45d62752b8cc5fe2e83bce377 not found: ID does not exist"
Nov 03 08:23:52.971682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.971629    6430 scope.go:122] "RemoveContainer" containerID="c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63"
Nov 03 08:23:52.983958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.983934    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"]
Nov 03 08:23:52.984458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.984438    6430 scope.go:122] "RemoveContainer" containerID="c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63"
Nov 03 08:23:52.984755 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:52.984736    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63\": container with ID starting with c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63 not found: ID does not exist" containerID="c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63"
Nov 03 08:23:52.984848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.984761    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63"} err="failed to get container status \"c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63\": rpc error: code = NotFound desc = could not find container \"c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63\": container with ID starting with c77f74550273efed44819945aaad63cab20373039afc1e06de0d4f502783be63 not found: ID does not exist"
Nov 03 08:23:52.987760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.987735    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-616e46/test-model-616e46-kserve-7787899fd7-jftrb"]
Nov 03 08:23:52.995021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:52.994995    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"]
Nov 03 08:23:53.000941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.000917    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e0f576-f658f96bc-bm2p9"]
Nov 03 08:23:53.265259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.265220    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" podUID="31bab6fc-0528-47ae-836e-743075206e82" containerName="main" containerID="cri-o://71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d" gracePeriod=45
Nov 03 08:23:53.584767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.584742    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.707619    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31bab6fc-0528-47ae-836e-743075206e82-tls-certs\" (UniqueName: \"kubernetes.io/secret/31bab6fc-0528-47ae-836e-743075206e82-tls-certs\") pod \"31bab6fc-0528-47ae-836e-743075206e82\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") "
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.708034    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-dshm\") pod \"31bab6fc-0528-47ae-836e-743075206e82\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") "
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.708140    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-model-cache\") pod \"31bab6fc-0528-47ae-836e-743075206e82\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") "
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.708579    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-tmp-dir\") pod \"31bab6fc-0528-47ae-836e-743075206e82\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") "
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.708822    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bab6fc-0528-47ae-836e-743075206e82-kube-api-access-smnsv\" (UniqueName: \"kubernetes.io/projected/31bab6fc-0528-47ae-836e-743075206e82-kube-api-access-smnsv\") pod \"31bab6fc-0528-47ae-836e-743075206e82\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") "
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.708889    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-home\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-home\") pod \"31bab6fc-0528-47ae-836e-743075206e82\" (UID: \"31bab6fc-0528-47ae-836e-743075206e82\") "
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.710176    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-tmp-dir" pod "31bab6fc-0528-47ae-836e-743075206e82" (UID: "31bab6fc-0528-47ae-836e-743075206e82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.710889    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-home" pod "31bab6fc-0528-47ae-836e-743075206e82" (UID: "31bab6fc-0528-47ae-836e-743075206e82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.712587    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-model-cache" pod "31bab6fc-0528-47ae-836e-743075206e82" (UID: "31bab6fc-0528-47ae-836e-743075206e82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.713045    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.713083    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:53.716102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.713282    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:53.746137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.743348    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-dshm" pod "31bab6fc-0528-47ae-836e-743075206e82" (UID: "31bab6fc-0528-47ae-836e-743075206e82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:53.746137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.743780    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bab6fc-0528-47ae-836e-743075206e82-tls-certs" pod "31bab6fc-0528-47ae-836e-743075206e82" (UID: "31bab6fc-0528-47ae-836e-743075206e82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:53.756670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.756633    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bab6fc-0528-47ae-836e-743075206e82-kube-api-access-smnsv" pod "31bab6fc-0528-47ae-836e-743075206e82" (UID: "31bab6fc-0528-47ae-836e-743075206e82"). InnerVolumeSpecName "kube-api-access-smnsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:53.814505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.814442    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smnsv\" (UniqueName: \"kubernetes.io/projected/31bab6fc-0528-47ae-836e-743075206e82-kube-api-access-smnsv\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:53.814505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.814466    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bab6fc-0528-47ae-836e-743075206e82-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:53.814505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.814478    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:53.814505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.814488    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bab6fc-0528-47ae-836e-743075206e82-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:53.972508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.972473    6430 generic.go:356] "Generic (PLEG): container finished" podID="31bab6fc-0528-47ae-836e-743075206e82" containerID="71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d" exitCode=0
Nov 03 08:23:53.972688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.972543    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"
Nov 03 08:23:53.972688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.972551    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" event={"ID":"31bab6fc-0528-47ae-836e-743075206e82","Type":"ContainerDied","Data":"71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d"}
Nov 03 08:23:53.972688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.972589    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" event={"ID":"31bab6fc-0528-47ae-836e-743075206e82","Type":"ContainerDied","Data":"898dc453559c2facbc12135f1c0a737ec1e724db2e06f8ae9f62c27f47eec669"}
Nov 03 08:23:53.972688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.972609    6430 scope.go:122] "RemoveContainer" containerID="71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d"
Nov 03 08:23:53.984610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.984589    6430 scope.go:122] "RemoveContainer" containerID="71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d"
Nov 03 08:23:53.989350 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:53.989319    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d\": container with ID starting with 71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d not found: ID does not exist" containerID="71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d"
Nov 03 08:23:53.989490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:53.989358    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d"} err="failed to get container status \"71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d\": rpc error: code = NotFound desc = could not find container \"71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d\": container with ID starting with 71242e67ef13e20034e93b36c681933b90565adb5defc88a3d2f5c98a0e3ad4d not found: ID does not exist"
Nov 03 08:23:54.005533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:54.004547    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"]
Nov 03 08:23:54.008504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:54.008483    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl"]
Nov 03 08:23:54.512772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:54.512692    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b8a152/test-model-b8a152-kserve-9965564cc-jjqcl" podUID="31bab6fc-0528-47ae-836e-743075206e82" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:23:54.531098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:54.526831    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:54.640552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:54.640517    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bab6fc-0528-47ae-836e-743075206e82" path="/var/lib/kubelet/pods/31bab6fc-0528-47ae-836e-743075206e82/volumes"
Nov 03 08:23:54.641138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:54.641050    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" path="/var/lib/kubelet/pods/7ee9f9ca-9623-4300-9b04-6c3ba857841d/volumes"
Nov 03 08:23:54.641674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:54.641656    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fd0430-b072-458b-91fa-8a1459789c95" path="/var/lib/kubelet/pods/89fd0430-b072-458b-91fa-8a1459789c95/volumes"
Nov 03 08:23:55.352237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:55.352191    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:55.978007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:55.977963    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:56.114163 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:56.114114    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:56.230862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:56.230775    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:56.636246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:56.636157    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:56.757031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:56.757004    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:56.909516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:56.907882    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:57.275387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:57.275352    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:57.914133 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:57.914096    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:58.378477 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:23:58.378433    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:23:58.750557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:58.750526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:23:59.492788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:23:59.492744    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:00.251128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:00.251088    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:01.170833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:01.170802    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:01.957217 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:01.957183    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:02.378054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.377975    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:02.486701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.486664    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:02.647464 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.647371    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"]
Nov 03 08:24:02.649475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.649089    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" containerID="cri-o://321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8" gracePeriod=30
Nov 03 08:24:02.705941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.705862    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"]
Nov 03 08:24:02.706131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.706034    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" containerID="cri-o://8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c" gracePeriod=30
Nov 03 08:24:02.708160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.708108    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:02.821821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.821779    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:02.955087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:02.950080    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:03.071088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:03.063537    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:03.342273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:03.342187    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:03.498494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:03.498459    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:03.498938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:03.498528    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:03.703590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:03.703518    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:04.441000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.440966    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:04.561047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.561001    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:04.681894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.681860    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:04.857862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.857767    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"]
Nov 03 08:24:04.858466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.858043    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy" containerID="cri-o://79d9fd01e53ecf240503776f1e6bc372d5f5f6c6db474a9bf19086258c49923f" gracePeriod=30
Nov 03 08:24:04.863396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.863368    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"]
Nov 03 08:24:04.863681 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.863636    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" containerID="cri-o://69a9a82ee685a1c37129e135e368afe0c89ae2256285a534c6349262c5e3dc16" gracePeriod=30
Nov 03 08:24:04.898977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:04.898944    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:05.264186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:05.264154    6430 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:05.264373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:05.264226    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:05.373438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:05.373405    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:06.099257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:06.099213    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:06.630466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:06.628741    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"]
Nov 03 08:24:06.630466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:06.628968    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h" podUID="8857534d-237b-4ff9-bf5b-b18cf35727f8" containerName="maas-api" containerID="cri-o://d7fb7cd9dac8425ba7546470660f04fd829c6873cffd8540d5e222288e5c1c62" gracePeriod=30
Nov 03 08:24:06.685741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:06.685697    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:07.176203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.176097    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:07.259630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.259593    6430 generic.go:356] "Generic (PLEG): container finished" podID="8857534d-237b-4ff9-bf5b-b18cf35727f8" containerID="d7fb7cd9dac8425ba7546470660f04fd829c6873cffd8540d5e222288e5c1c62" exitCode=0
Nov 03 08:24:07.259964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.259941    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h" event={"ID":"8857534d-237b-4ff9-bf5b-b18cf35727f8","Type":"ContainerDied","Data":"d7fb7cd9dac8425ba7546470660f04fd829c6873cffd8540d5e222288e5c1c62"}
Nov 03 08:24:07.286339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.286315    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:24:07.288734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.288705    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:07.404680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.404646    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:07.423884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.423854    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-metrics-certs\") pod \"8857534d-237b-4ff9-bf5b-b18cf35727f8\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") "
Nov 03 08:24:07.424127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.423964    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8857534d-237b-4ff9-bf5b-b18cf35727f8-kube-api-access-4zmm4\" (UniqueName: \"kubernetes.io/projected/8857534d-237b-4ff9-bf5b-b18cf35727f8-kube-api-access-4zmm4\") pod \"8857534d-237b-4ff9-bf5b-b18cf35727f8\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") "
Nov 03 08:24:07.424127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.423984    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls\") pod \"8857534d-237b-4ff9-bf5b-b18cf35727f8\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") "
Nov 03 08:24:07.424127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.424027    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8857534d-237b-4ff9-bf5b-b18cf35727f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8857534d-237b-4ff9-bf5b-b18cf35727f8-openshift-service-ca\") pod \"8857534d-237b-4ff9-bf5b-b18cf35727f8\" (UID: \"8857534d-237b-4ff9-bf5b-b18cf35727f8\") "
Nov 03 08:24:07.424557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.424460    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8857534d-237b-4ff9-bf5b-b18cf35727f8-openshift-service-ca" pod "8857534d-237b-4ff9-bf5b-b18cf35727f8" (UID: "8857534d-237b-4ff9-bf5b-b18cf35727f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:07.424852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.424827    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8857534d-237b-4ff9-bf5b-b18cf35727f8-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:07.426608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.426554    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-metrics-certs" pod "8857534d-237b-4ff9-bf5b-b18cf35727f8" (UID: "8857534d-237b-4ff9-bf5b-b18cf35727f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:07.427171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.427155    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls" pod "8857534d-237b-4ff9-bf5b-b18cf35727f8" (UID: "8857534d-237b-4ff9-bf5b-b18cf35727f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:07.427267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.427224    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8857534d-237b-4ff9-bf5b-b18cf35727f8-kube-api-access-4zmm4" pod "8857534d-237b-4ff9-bf5b-b18cf35727f8" (UID: "8857534d-237b-4ff9-bf5b-b18cf35727f8"). InnerVolumeSpecName "kube-api-access-4zmm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:07.526227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.526202    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zmm4\" (UniqueName: \"kubernetes.io/projected/8857534d-237b-4ff9-bf5b-b18cf35727f8-kube-api-access-4zmm4\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:07.526227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.526227    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:07.526401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:07.526236    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8857534d-237b-4ff9-bf5b-b18cf35727f8-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.018109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.018049    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:24:08.094598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.094555    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:08.095236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095209    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-certs\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095329    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-socket\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095361    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-credential-socket\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-token\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-token\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095586    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35a857c-1664-4657-ab11-9b5ee33baa24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a857c-1664-4657-ab11-9b5ee33baa24-istiod-ca-cert\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095626    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-podinfo\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095676    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-data\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095706    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-kube-api-access-km4bz\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-kube-api-access-km4bz\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.095870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.095855    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-envoy\") pod \"e35a857c-1664-4657-ab11-9b5ee33baa24\" (UID: \"e35a857c-1664-4657-ab11-9b5ee33baa24\") "
Nov 03 08:24:08.099592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.099562    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-envoy" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.099838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.099803    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-certs" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.100146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.100124    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-socket" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.100376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.100358    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-credential-socket" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.101313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.101288    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35a857c-1664-4657-ab11-9b5ee33baa24-istiod-ca-cert" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:08.102305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.102278    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-data" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.103256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.103229    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-token" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.104183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.104152    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-podinfo" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:08.113255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.113218    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-kube-api-access-km4bz" pod "e35a857c-1664-4657-ab11-9b5ee33baa24" (UID: "e35a857c-1664-4657-ab11-9b5ee33baa24"). InnerVolumeSpecName "kube-api-access-km4bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.125158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.125132    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:24:08.197494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197405    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-socket\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.197960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197518    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-data\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.197960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197680    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-envoy\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.197960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197711    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-certs\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.197960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197706    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-socket" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.197960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197743    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-istio-token\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-istio-token\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.197960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197832    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-kube-api-access-t8vzz\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-kube-api-access-t8vzz\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.197960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.197892    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43b9ca2-2706-4449-8102-55b7b146379e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b9ca2-2706-4449-8102-55b7b146379e-istiod-ca-cert\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.198383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198044    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f43b9ca2-2706-4449-8102-55b7b146379e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b9ca2-2706-4449-8102-55b7b146379e-istio-podinfo\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.198383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198097    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-credential-socket\") pod \"f43b9ca2-2706-4449-8102-55b7b146379e\" (UID: \"f43b9ca2-2706-4449-8102-55b7b146379e\") "
Nov 03 08:24:08.198383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198240    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-data" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.198383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198361    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-certs" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.198580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198545    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-credential-socket" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.198641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198585    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43b9ca2-2706-4449-8102-55b7b146379e-istiod-ca-cert" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:08.198907 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198882    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.198999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198911    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.198999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198924    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.198999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198937    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.198999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198950    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.198999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198963    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.198999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198979    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.198999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.198994    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a857c-1664-4657-ab11-9b5ee33baa24-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.199579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.199008    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.199579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.199023    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.199579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.199238    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4bz\" (UniqueName: \"kubernetes.io/projected/e35a857c-1664-4657-ab11-9b5ee33baa24-kube-api-access-km4bz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.199579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.199254    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b9ca2-2706-4449-8102-55b7b146379e-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.199579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.199269    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a857c-1664-4657-ab11-9b5ee33baa24-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.199579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.199281    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.201000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.200964    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-kube-api-access-t8vzz" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "kube-api-access-t8vzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.201161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.201135    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-istio-token" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.201436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.201418    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-envoy" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.201694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.201678    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f43b9ca2-2706-4449-8102-55b7b146379e-istio-podinfo" pod "f43b9ca2-2706-4449-8102-55b7b146379e" (UID: "f43b9ca2-2706-4449-8102-55b7b146379e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:08.277486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.277453    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"
Nov 03 08:24:08.277693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.277476    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h" event={"ID":"8857534d-237b-4ff9-bf5b-b18cf35727f8","Type":"ContainerDied","Data":"8abf22f4146bfe9525349f79166f6157cef8c1a27749bb7806c3b874c5daad25"}
Nov 03 08:24:08.277693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.277558    6430 scope.go:122] "RemoveContainer" containerID="d7fb7cd9dac8425ba7546470660f04fd829c6873cffd8540d5e222288e5c1c62"
Nov 03 08:24:08.287339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.285781    6430 generic.go:356] "Generic (PLEG): container finished" podID="f43b9ca2-2706-4449-8102-55b7b146379e" containerID="8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c" exitCode=0
Nov 03 08:24:08.287339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.285862    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" event={"ID":"f43b9ca2-2706-4449-8102-55b7b146379e","Type":"ContainerDied","Data":"8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c"}
Nov 03 08:24:08.287339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.285942    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj" event={"ID":"f43b9ca2-2706-4449-8102-55b7b146379e","Type":"ContainerDied","Data":"dc8d870362959e480aed71f00a272b99d8327df6fa8b2f0a0d5fe7ad41c1741c"}
Nov 03 08:24:08.287339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.285874    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"
Nov 03 08:24:08.289771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.288765    6430 generic.go:356] "Generic (PLEG): container finished" podID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerID="321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8" exitCode=0
Nov 03 08:24:08.289771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.288830    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" event={"ID":"e35a857c-1664-4657-ab11-9b5ee33baa24","Type":"ContainerDied","Data":"321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8"}
Nov 03 08:24:08.289771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.288872    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk" event={"ID":"e35a857c-1664-4657-ab11-9b5ee33baa24","Type":"ContainerDied","Data":"b10ce251b6b9c3861fd7b9af7700158c34c59860847807fdeec686e0e9a177de"}
Nov 03 08:24:08.289771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.288950    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"
Nov 03 08:24:08.295494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.295242    6430 scope.go:122] "RemoveContainer" containerID="8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c"
Nov 03 08:24:08.300668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.300643    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vzz\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-kube-api-access-t8vzz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.300790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.300672    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b9ca2-2706-4449-8102-55b7b146379e-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.300790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.300688    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b9ca2-2706-4449-8102-55b7b146379e-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.300790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.300701    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b9ca2-2706-4449-8102-55b7b146379e-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.307500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.307476    6430 scope.go:122] "RemoveContainer" containerID="8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c"
Nov 03 08:24:08.308337 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:08.308313    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c\": container with ID starting with 8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c not found: ID does not exist" containerID="8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c"
Nov 03 08:24:08.308438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.308345    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c"} err="failed to get container status \"8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c\": rpc error: code = NotFound desc = could not find container \"8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c\": container with ID starting with 8d712334638ff4340bc941c0fb36ad9ee5be6217b3c3b8f8973f613b4a178c6c not found: ID does not exist"
Nov 03 08:24:08.308438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.308370    6430 scope.go:122] "RemoveContainer" containerID="321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8"
Nov 03 08:24:08.314506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.314482    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"]
Nov 03 08:24:08.320807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.320784    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-76ef13-77fc7fb78c-st96h"]
Nov 03 08:24:08.322440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.322424    6430 scope.go:122] "RemoveContainer" containerID="321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8"
Nov 03 08:24:08.322763 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:08.322742    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8\": container with ID starting with 321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8 not found: ID does not exist" containerID="321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8"
Nov 03 08:24:08.322850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.322770    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8"} err="failed to get container status \"321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8\": rpc error: code = NotFound desc = could not find container \"321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8\": container with ID starting with 321a7f25ca8018099cc77aa4cf3370947a5a04016ac081e652f66984a7e441e8 not found: ID does not exist"
Nov 03 08:24:08.331844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.331822    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"]
Nov 03 08:24:08.335322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.335302    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-gskdj"]
Nov 03 08:24:08.342971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.342933    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"]
Nov 03 08:24:08.346734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.346713    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-6s6rk"]
Nov 03 08:24:08.619396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.619325    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8857534d-237b-4ff9-bf5b-b18cf35727f8" path="/var/lib/kubelet/pods/8857534d-237b-4ff9-bf5b-b18cf35727f8/volumes"
Nov 03 08:24:08.619833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.619819    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" path="/var/lib/kubelet/pods/e35a857c-1664-4657-ab11-9b5ee33baa24/volumes"
Nov 03 08:24:08.620269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.620256    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" path="/var/lib/kubelet/pods/f43b9ca2-2706-4449-8102-55b7b146379e/volumes"
Nov 03 08:24:08.851452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:08.851425    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:09.108893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:09.108856    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:09.109048 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:09.108914    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:09.700582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:09.700556    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:09.848084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:09.834198    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:10.069795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.069757    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"]
Nov 03 08:24:10.070030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.070003    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy" containerID="cri-o://09b6aa980cc2fd1a8559afa4070f259f58d167561f4475ace4f58d09b6fd03da" gracePeriod=30
Nov 03 08:24:10.072424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.072398    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"]
Nov 03 08:24:10.072636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.072608    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" containerID="cri-o://cb15341d1b72c99408df5ea6b7a6df10d882e5f309330d0655f9cbeb14a95b33" gracePeriod=30
Nov 03 08:24:10.122490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.122444    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:10.338561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.335176    6430 generic.go:356] "Generic (PLEG): container finished" podID="e57e36a8-937e-44f0-b243-a95a341901a0" containerID="69a9a82ee685a1c37129e135e368afe0c89ae2256285a534c6349262c5e3dc16" exitCode=0
Nov 03 08:24:10.338561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.335401    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" event={"ID":"e57e36a8-937e-44f0-b243-a95a341901a0","Type":"ContainerDied","Data":"69a9a82ee685a1c37129e135e368afe0c89ae2256285a534c6349262c5e3dc16"}
Nov 03 08:24:10.338561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.335456    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h" event={"ID":"e57e36a8-937e-44f0-b243-a95a341901a0","Type":"ContainerDied","Data":"9ce5723efd4db42fae045b2915d2f6f6d58a6d81c093eb2ff6ff14212dd342aa"}
Nov 03 08:24:10.338561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.335480    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ce5723efd4db42fae045b2915d2f6f6d58a6d81c093eb2ff6ff14212dd342aa"
Nov 03 08:24:10.338561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.337932    6430 generic.go:356] "Generic (PLEG): container finished" podID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerID="79d9fd01e53ecf240503776f1e6bc372d5f5f6c6db474a9bf19086258c49923f" exitCode=0
Nov 03 08:24:10.338561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.338002    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" event={"ID":"a22dcaae-af57-4cd0-b304-fa20192b9a62","Type":"ContainerDied","Data":"79d9fd01e53ecf240503776f1e6bc372d5f5f6c6db474a9bf19086258c49923f"}
Nov 03 08:24:10.347625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.347595    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375327    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-istio-token\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-istio-token\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375496    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-credential-socket\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375593    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-socket\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375647    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-certs\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375728    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-envoy\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375766    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e57e36a8-937e-44f0-b243-a95a341901a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57e36a8-937e-44f0-b243-a95a341901a0-istio-podinfo\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375838    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57e36a8-937e-44f0-b243-a95a341901a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57e36a8-937e-44f0-b243-a95a341901a0-istiod-ca-cert\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375922    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-data\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.376300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.375998    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-kube-api-access-4vjqp\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-kube-api-access-4vjqp\") pod \"e57e36a8-937e-44f0-b243-a95a341901a0\" (UID: \"e57e36a8-937e-44f0-b243-a95a341901a0\") "
Nov 03 08:24:10.377859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.376794    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-certs" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.379513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.379376    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-credential-socket" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.379721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.379698    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-data" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.379845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.379756    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57e36a8-937e-44f0-b243-a95a341901a0-istiod-ca-cert" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:10.381047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.381018    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-socket" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.402239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.400818    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-envoy" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.402239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.401232    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e57e36a8-937e-44f0-b243-a95a341901a0-istio-podinfo" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:10.402239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.401363    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-istio-token" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:10.402535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.402494    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-kube-api-access-4vjqp" pod "e57e36a8-937e-44f0-b243-a95a341901a0" (UID: "e57e36a8-937e-44f0-b243-a95a341901a0"). InnerVolumeSpecName "kube-api-access-4vjqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:10.478430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478392    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478433    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478458    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478472    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478486    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478498    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57e36a8-937e-44f0-b243-a95a341901a0-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478508    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57e36a8-937e-44f0-b243-a95a341901a0-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478522    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57e36a8-937e-44f0-b243-a95a341901a0-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.478624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.478533    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vjqp\" (UniqueName: \"kubernetes.io/projected/e57e36a8-937e-44f0-b243-a95a341901a0-kube-api-access-4vjqp\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.595810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.595781    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:24:10.681304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681273    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-token\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-token\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681328    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-kube-api-access-hqk5w\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-kube-api-access-hqk5w\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681353    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-podinfo\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681433    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22dcaae-af57-4cd0-b304-fa20192b9a62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22dcaae-af57-4cd0-b304-fa20192b9a62-istiod-ca-cert\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681505    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-data\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681644    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-envoy\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681685    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-credential-socket\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681720    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-socket\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681771    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-certs\") pod \"a22dcaae-af57-4cd0-b304-fa20192b9a62\" (UID: \"a22dcaae-af57-4cd0-b304-fa20192b9a62\") "
Nov 03 08:24:10.681923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681856    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-data" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.681923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.681895    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22dcaae-af57-4cd0-b304-fa20192b9a62-istiod-ca-cert" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:10.682138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682111    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-credential-socket" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.682389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682261    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-certs" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.682389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682277    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-socket" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.682550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682524    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22dcaae-af57-4cd0-b304-fa20192b9a62-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.682550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682553    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.682550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682567    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.682550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682583    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.682550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.682600    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.683862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.683834    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-token" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:10.683953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.683934    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-envoy" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.684025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.684008    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-podinfo" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:10.684474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.684456    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-kube-api-access-hqk5w" pod "a22dcaae-af57-4cd0-b304-fa20192b9a62" (UID: "a22dcaae-af57-4cd0-b304-fa20192b9a62"). InnerVolumeSpecName "kube-api-access-hqk5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:10.784038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.784007    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.784038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.784036    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.784428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.784046    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqk5w\" (UniqueName: \"kubernetes.io/projected/a22dcaae-af57-4cd0-b304-fa20192b9a62-kube-api-access-hqk5w\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.784428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.784084    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22dcaae-af57-4cd0-b304-fa20192b9a62-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.925210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:10.925178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:11.346658 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.346628    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"
Nov 03 08:24:11.346868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.346663    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645" event={"ID":"a22dcaae-af57-4cd0-b304-fa20192b9a62","Type":"ContainerDied","Data":"0767e028dcaf55e86cce179c4561a9432a09395ce1029c6e38bd8faa5798dfef"}
Nov 03 08:24:11.346868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.346730    6430 scope.go:122] "RemoveContainer" containerID="79d9fd01e53ecf240503776f1e6bc372d5f5f6c6db474a9bf19086258c49923f"
Nov 03 08:24:11.347000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.346891    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"
Nov 03 08:24:11.371277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.371237    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"]
Nov 03 08:24:11.374791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.374770    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-6sz4h"]
Nov 03 08:24:11.383538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.383517    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"]
Nov 03 08:24:11.386415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.386395    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-cf645"]
Nov 03 08:24:11.743313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.743280    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:11.743468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.743332    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:11.910913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:11.910873    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:12.191277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:12.191248    6430 scope.go:122] "RemoveContainer" containerID="69a9a82ee685a1c37129e135e368afe0c89ae2256285a534c6349262c5e3dc16"
Nov 03 08:24:12.202233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:12.202211    6430 scope.go:122] "RemoveContainer" containerID="ac5d2f23d2e52733f714494c9c02dc13eea515b49ae6e525b9a82b1d4d0aa8f0"
Nov 03 08:24:12.619796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:12.619719    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" path="/var/lib/kubelet/pods/a22dcaae-af57-4cd0-b304-fa20192b9a62/volumes"
Nov 03 08:24:12.620260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:12.620244    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" path="/var/lib/kubelet/pods/e57e36a8-937e-44f0-b243-a95a341901a0/volumes"
Nov 03 08:24:12.755902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:12.755854    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:13.341225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:13.341190    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:13.441847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:13.441812    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"]
Nov 03 08:24:13.492979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:13.492945    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:13.694498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:13.694456    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"]
Nov 03 08:24:13.694743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:13.694719    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy" containerID="cri-o://2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642" gracePeriod=30
Nov 03 08:24:13.756671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:13.756638    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:14.188559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:14.188507    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:14.543516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:14.543434    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:15.109726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.109694    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"]
Nov 03 08:24:15.109947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.109921    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls" podUID="4bde8834-c6f6-44d8-8df3-542c155b7315" containerName="maas-api" containerID="cri-o://8d020769d116d1030011c1c2321b3b9dcbbb7ce28d1b92c1feb2678066d8c03b" gracePeriod=30
Nov 03 08:24:15.159280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.159246    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:15.520095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.519871    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:15.580480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.579364    6430 generic.go:356] "Generic (PLEG): container finished" podID="83537c3e-e723-4150-a413-c817d9630222" containerID="09b6aa980cc2fd1a8559afa4070f259f58d167561f4475ace4f58d09b6fd03da" exitCode=0
Nov 03 08:24:15.580480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.579575    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" event={"ID":"83537c3e-e723-4150-a413-c817d9630222","Type":"ContainerDied","Data":"09b6aa980cc2fd1a8559afa4070f259f58d167561f4475ace4f58d09b6fd03da"}
Nov 03 08:24:15.614396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.614177    6430 generic.go:356] "Generic (PLEG): container finished" podID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerID="cb15341d1b72c99408df5ea6b7a6df10d882e5f309330d0655f9cbeb14a95b33" exitCode=0
Nov 03 08:24:15.614530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.614400    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" event={"ID":"5eb640fd-a708-4f4f-8607-ec9e414ad41a","Type":"ContainerDied","Data":"cb15341d1b72c99408df5ea6b7a6df10d882e5f309330d0655f9cbeb14a95b33"}
Nov 03 08:24:15.622495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.621620    6430 generic.go:356] "Generic (PLEG): container finished" podID="4bde8834-c6f6-44d8-8df3-542c155b7315" containerID="8d020769d116d1030011c1c2321b3b9dcbbb7ce28d1b92c1feb2678066d8c03b" exitCode=0
Nov 03 08:24:15.622495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.621714    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls" event={"ID":"4bde8834-c6f6-44d8-8df3-542c155b7315","Type":"ContainerDied","Data":"8d020769d116d1030011c1c2321b3b9dcbbb7ce28d1b92c1feb2678066d8c03b"}
Nov 03 08:24:15.633823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.633753    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:15.731155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.731132    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:24:15.809988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.809955    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:24:15.815731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.815707    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.832952    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bde8834-c6f6-44d8-8df3-542c155b7315-kube-api-access-zwcdw\" (UniqueName: \"kubernetes.io/projected/4bde8834-c6f6-44d8-8df3-542c155b7315-kube-api-access-zwcdw\") pod \"4bde8834-c6f6-44d8-8df3-542c155b7315\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") "
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.833086    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-maas-api-tls\") pod \"4bde8834-c6f6-44d8-8df3-542c155b7315\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") "
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.833228    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bde8834-c6f6-44d8-8df3-542c155b7315-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde8834-c6f6-44d8-8df3-542c155b7315-openshift-service-ca\") pod \"4bde8834-c6f6-44d8-8df3-542c155b7315\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") "
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.833339    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-metrics-certs\") pod \"4bde8834-c6f6-44d8-8df3-542c155b7315\" (UID: \"4bde8834-c6f6-44d8-8df3-542c155b7315\") "
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.834674    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bde8834-c6f6-44d8-8df3-542c155b7315-openshift-service-ca" pod "4bde8834-c6f6-44d8-8df3-542c155b7315" (UID: "4bde8834-c6f6-44d8-8df3-542c155b7315"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.836359    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-maas-api-tls" pod "4bde8834-c6f6-44d8-8df3-542c155b7315" (UID: "4bde8834-c6f6-44d8-8df3-542c155b7315"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.837589    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-metrics-certs" pod "4bde8834-c6f6-44d8-8df3-542c155b7315" (UID: "4bde8834-c6f6-44d8-8df3-542c155b7315"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:15.839650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.837903    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bde8834-c6f6-44d8-8df3-542c155b7315-kube-api-access-zwcdw" pod "4bde8834-c6f6-44d8-8df3-542c155b7315" (UID: "4bde8834-c6f6-44d8-8df3-542c155b7315"). InnerVolumeSpecName "kube-api-access-zwcdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:15.934791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.934761    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-data\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.934791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.934797    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83537c3e-e723-4150-a413-c817d9630222-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83537c3e-e723-4150-a413-c817d9630222-istio-podinfo\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.934989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.934832    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-envoy\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.934989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.934910    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-istio-token\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-istio-token\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.934989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.934941    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-podinfo\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.934989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.934980    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-token\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.935243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935019    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-credential-socket\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.935243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935095    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-data" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.935243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935166    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-certs\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.935243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935208    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istiod-ca-cert\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.935243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935236    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-certs\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.935535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935295    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-kube-api-access-zc4r4\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-kube-api-access-zc4r4\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.935535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935369    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83537c3e-e723-4150-a413-c817d9630222-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83537c3e-e723-4150-a413-c817d9630222-istiod-ca-cert\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.935684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935643    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-credential-socket" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.935862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935839    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-certs" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.936024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.935975    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istiod-ca-cert" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:15.936226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936025    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-credential-socket\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.936226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936132    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-kube-api-access-rj7zm\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-kube-api-access-rj7zm\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.936226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936181    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-socket\") pod \"83537c3e-e723-4150-a413-c817d9630222\" (UID: \"83537c3e-e723-4150-a413-c817d9630222\") "
Nov 03 08:24:15.936226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936212    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-socket\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.936445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936265    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-credential-socket" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.936445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936334    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-data\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.936445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936364    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-envoy\") pod \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\" (UID: \"5eb640fd-a708-4f4f-8607-ec9e414ad41a\") "
Nov 03 08:24:15.936905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.936879    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-socket" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.937463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937434    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcdw\" (UniqueName: \"kubernetes.io/projected/4bde8834-c6f6-44d8-8df3-542c155b7315-kube-api-access-zwcdw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937460    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937474    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937488    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937500    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde8834-c6f6-44d8-8df3-542c155b7315-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937513    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937527    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937540    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bde8834-c6f6-44d8-8df3-542c155b7315-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937554    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.937638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937566    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.938769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.937764    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-certs" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.938769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.938121    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-socket" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.938769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.938424    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83537c3e-e723-4150-a413-c817d9630222-istiod-ca-cert" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:15.938769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.938560    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-data" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.938962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.938809    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-envoy" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.938962 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.938869    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83537c3e-e723-4150-a413-c817d9630222-istio-podinfo" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:15.939031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.938979    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-kube-api-access-zc4r4" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "kube-api-access-zc4r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:15.939426 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.939403    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-token" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:15.939511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.939459    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-istio-token" pod "83537c3e-e723-4150-a413-c817d9630222" (UID: "83537c3e-e723-4150-a413-c817d9630222"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:15.939844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.939819    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-podinfo" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:15.940262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.940240    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-kube-api-access-rj7zm" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "kube-api-access-rj7zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:15.940499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:15.940475    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-envoy" pod "5eb640fd-a708-4f4f-8607-ec9e414ad41a" (UID: "5eb640fd-a708-4f4f-8607-ec9e414ad41a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.030933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.030857    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:16.038800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038756    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.038800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038800    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038815    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038829    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038842    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc4r4\" (UniqueName: \"kubernetes.io/projected/83537c3e-e723-4150-a413-c817d9630222-kube-api-access-zc4r4\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038854    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83537c3e-e723-4150-a413-c817d9630222-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038867    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7zm\" (UniqueName: \"kubernetes.io/projected/5eb640fd-a708-4f4f-8607-ec9e414ad41a-kube-api-access-rj7zm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038880    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038895    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038907    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb640fd-a708-4f4f-8607-ec9e414ad41a-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038919    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83537c3e-e723-4150-a413-c817d9630222-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.039002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.038930    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83537c3e-e723-4150-a413-c817d9630222-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.634000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.633975    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"
Nov 03 08:24:16.634473 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.634004    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls" event={"ID":"4bde8834-c6f6-44d8-8df3-542c155b7315","Type":"ContainerDied","Data":"19d058a805805d96b465248802c44e12c2f402a6fea13400d71face8f4c9ec31"}
Nov 03 08:24:16.634473 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.634110    6430 scope.go:122] "RemoveContainer" containerID="8d020769d116d1030011c1c2321b3b9dcbbb7ce28d1b92c1feb2678066d8c03b"
Nov 03 08:24:16.635683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.635666    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"
Nov 03 08:24:16.635800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.635729    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5" event={"ID":"83537c3e-e723-4150-a413-c817d9630222","Type":"ContainerDied","Data":"2ee6991ae2d56fcb389a9f0901edff59b01de8db1cb17d15e0973dbeb208305d"}
Nov 03 08:24:16.640486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.640441    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj" event={"ID":"5eb640fd-a708-4f4f-8607-ec9e414ad41a","Type":"ContainerDied","Data":"bacd809281635656d5dd2b6e83c8cacf61bbff9c3fc07181fa65edb579dd7a40"}
Nov 03 08:24:16.640579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.640468    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"
Nov 03 08:24:16.646459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.646416    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:16.648541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.648522    6430 scope.go:122] "RemoveContainer" containerID="09b6aa980cc2fd1a8559afa4070f259f58d167561f4475ace4f58d09b6fd03da"
Nov 03 08:24:16.747965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.747922    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"]
Nov 03 08:24:16.753197 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.752600    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-4w9r5"]
Nov 03 08:24:16.757127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.757054    6430 scope.go:122] "RemoveContainer" containerID="cb15341d1b72c99408df5ea6b7a6df10d882e5f309330d0655f9cbeb14a95b33"
Nov 03 08:24:16.757633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.757610    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"]
Nov 03 08:24:16.762251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.762227    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-616e46-78c78c5fdf-8psls"]
Nov 03 08:24:16.766642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.766592    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"]
Nov 03 08:24:16.770717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:16.770698    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zmmnj"]
Nov 03 08:24:17.357906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:17.357874    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:18.078896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.078863    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:18.442471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.442422    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j" podUID="d0fd224e-e7a1-46be-89cf-3fae1f1120e2" containerName="payload-processing" containerID="cri-o://f72271a77abd74d9927de01f733c9d303b94377e1687d0db7047d3990118ef08" gracePeriod=25
Nov 03 08:24:18.622432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.622400    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bde8834-c6f6-44d8-8df3-542c155b7315" path="/var/lib/kubelet/pods/4bde8834-c6f6-44d8-8df3-542c155b7315/volumes"
Nov 03 08:24:18.623202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.623030    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" path="/var/lib/kubelet/pods/5eb640fd-a708-4f4f-8607-ec9e414ad41a/volumes"
Nov 03 08:24:18.623766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.623742    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83537c3e-e723-4150-a413-c817d9630222" path="/var/lib/kubelet/pods/83537c3e-e723-4150-a413-c817d9630222/volumes"
Nov 03 08:24:18.662758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.662727    6430 generic.go:356] "Generic (PLEG): container finished" podID="d0fd224e-e7a1-46be-89cf-3fae1f1120e2" containerID="f72271a77abd74d9927de01f733c9d303b94377e1687d0db7047d3990118ef08" exitCode=0
Nov 03 08:24:18.662906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.662771    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j" event={"ID":"d0fd224e-e7a1-46be-89cf-3fae1f1120e2","Type":"ContainerDied","Data":"f72271a77abd74d9927de01f733c9d303b94377e1687d0db7047d3990118ef08"}
Nov 03 08:24:18.771011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.770981    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:18.835336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.835304    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:24:18.979913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.979882    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-kube-api-access-g7bcz\" (UniqueName: \"kubernetes.io/projected/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-kube-api-access-g7bcz\") pod \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") "
Nov 03 08:24:18.980036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.980015    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-plugins-config-volume\") pod \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\" (UID: \"d0fd224e-e7a1-46be-89cf-3fae1f1120e2\") "
Nov 03 08:24:18.980392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.980367    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-plugins-config-volume" pod "d0fd224e-e7a1-46be-89cf-3fae1f1120e2" (UID: "d0fd224e-e7a1-46be-89cf-3fae1f1120e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:18.980723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.980705    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:18.982210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.982187    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-kube-api-access-g7bcz" pod "d0fd224e-e7a1-46be-89cf-3fae1f1120e2" (UID: "d0fd224e-e7a1-46be-89cf-3fae1f1120e2"). InnerVolumeSpecName "kube-api-access-g7bcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:18.999262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:18.999238    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:24:19.081921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.081887    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-data\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.081921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.081925    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-kube-api-access-gnpbl\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-kube-api-access-gnpbl\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082097    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f954b829-91a3-4aee-925c-363f1cfa9824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f954b829-91a3-4aee-925c-363f1cfa9824-istio-podinfo\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082134    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f954b829-91a3-4aee-925c-363f1cfa9824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f954b829-91a3-4aee-925c-363f1cfa9824-istiod-ca-cert\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082169    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-data" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082299    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-certs\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082340    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-credential-socket\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082399    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-socket\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082434    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-istio-token\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-istio-token\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082480    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f954b829-91a3-4aee-925c-363f1cfa9824-istiod-ca-cert" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:19.082921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082497    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-envoy\") pod \"f954b829-91a3-4aee-925c-363f1cfa9824\" (UID: \"f954b829-91a3-4aee-925c-363f1cfa9824\") "
Nov 03 08:24:19.082921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082680    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-credential-socket" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.082921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082749    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-certs" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.082921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.082898    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-socket" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.083963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.083935    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.084117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.083968    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7bcz\" (UniqueName: \"kubernetes.io/projected/d0fd224e-e7a1-46be-89cf-3fae1f1120e2-kube-api-access-g7bcz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.084117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.083983    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.084117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.083998    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.084117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.084011    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.084117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.084024    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f954b829-91a3-4aee-925c-363f1cfa9824-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.085004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.084980    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f954b829-91a3-4aee-925c-363f1cfa9824-istio-podinfo" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:19.085313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.085286    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-istio-token" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:19.085547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.085526    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-kube-api-access-gnpbl" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "kube-api-access-gnpbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:19.085617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.085604    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-envoy" pod "f954b829-91a3-4aee-925c-363f1cfa9824" (UID: "f954b829-91a3-4aee-925c-363f1cfa9824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.185159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.185129    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.185159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.185153    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f954b829-91a3-4aee-925c-363f1cfa9824-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.185159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.185163    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnpbl\" (UniqueName: \"kubernetes.io/projected/f954b829-91a3-4aee-925c-363f1cfa9824-kube-api-access-gnpbl\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.185360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.185172    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f954b829-91a3-4aee-925c-363f1cfa9824-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.550573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.550528    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:19.672040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.672004    6430 generic.go:356] "Generic (PLEG): container finished" podID="f954b829-91a3-4aee-925c-363f1cfa9824" containerID="2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642" exitCode=0
Nov 03 08:24:19.672213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.672075    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"
Nov 03 08:24:19.672213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.672098    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" event={"ID":"f954b829-91a3-4aee-925c-363f1cfa9824","Type":"ContainerDied","Data":"2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642"}
Nov 03 08:24:19.672213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.672164    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt" event={"ID":"f954b829-91a3-4aee-925c-363f1cfa9824","Type":"ContainerDied","Data":"5e8c91ba54a82cd60277dc62231ad83af25fe090f2e6e2154140aea0e1b90cfd"}
Nov 03 08:24:19.672213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.672195    6430 scope.go:122] "RemoveContainer" containerID="2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642"
Nov 03 08:24:19.673812 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.673789    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j" event={"ID":"d0fd224e-e7a1-46be-89cf-3fae1f1120e2","Type":"ContainerDied","Data":"b75d609280f96c177cde807a7939893410a3228b7da80dfad9dbf9c30304a394"}
Nov 03 08:24:19.673959 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.673941    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"
Nov 03 08:24:19.685537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.685521    6430 scope.go:122] "RemoveContainer" containerID="2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642"
Nov 03 08:24:19.685820 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:19.685796    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642\": container with ID starting with 2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642 not found: ID does not exist" containerID="2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642"
Nov 03 08:24:19.685940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.685825    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642"} err="failed to get container status \"2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642\": rpc error: code = NotFound desc = could not find container \"2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642\": container with ID starting with 2ffec5608e83bbecfa1276eeff8379c0880a14b828f6680332ec11f7b7275642 not found: ID does not exist"
Nov 03 08:24:19.685940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.685842    6430 scope.go:122] "RemoveContainer" containerID="f72271a77abd74d9927de01f733c9d303b94377e1687d0db7047d3990118ef08"
Nov 03 08:24:19.700193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.700126    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"]
Nov 03 08:24:19.709026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.709001    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-nb9wt"]
Nov 03 08:24:19.712404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.712355    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"]
Nov 03 08:24:19.714693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:19.714672    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-616e46-5566d5856c-8zt8j"]
Nov 03 08:24:20.403247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:20.403216    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:20.618812 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:20.618777    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0fd224e-e7a1-46be-89cf-3fae1f1120e2" path="/var/lib/kubelet/pods/d0fd224e-e7a1-46be-89cf-3fae1f1120e2/volumes"
Nov 03 08:24:20.619271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:20.619257    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" path="/var/lib/kubelet/pods/f954b829-91a3-4aee-925c-363f1cfa9824/volumes"
Nov 03 08:24:21.061705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:21.061673    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:21.172797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:21.172757    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:21.997192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:21.997158    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:22.204537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.204504    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:22.311717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.311630    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:22.477620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.477575    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"]
Nov 03 08:24:22.477832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.477806    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy" containerID="cri-o://26e36941f422b7e1e8334db4d17477f201af1e587f7a2fc975aa655576a89b2f" gracePeriod=30
Nov 03 08:24:22.482427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.482360    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"]
Nov 03 08:24:22.482857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.482830    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy" containerID="cri-o://5db2038d101c0edbba162c3530431c6e87ad83ab1ae34e0ef1127c593a8ef1d2" gracePeriod=30
Nov 03 08:24:22.487550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.487517    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"]
Nov 03 08:24:22.487731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.487706    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy" containerID="cri-o://d1b62c7e3293ff58ec82c11b61d309ef11dd6d5bdf98138ec9c515b3e57835cb" gracePeriod=30
Nov 03 08:24:22.538141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.538111    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:22.865679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.865648    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:22.986052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:22.986017    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:23.985859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:23.985812    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:24.862725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:24.862695    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:25.685906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:25.685866    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:26.532947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:26.532917    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:27.246818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.246782    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:27.766950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.765888    6430 generic.go:356] "Generic (PLEG): container finished" podID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerID="d1b62c7e3293ff58ec82c11b61d309ef11dd6d5bdf98138ec9c515b3e57835cb" exitCode=0
Nov 03 08:24:27.766950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.765999    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" event={"ID":"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6","Type":"ContainerDied","Data":"d1b62c7e3293ff58ec82c11b61d309ef11dd6d5bdf98138ec9c515b3e57835cb"}
Nov 03 08:24:27.768506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.768356    6430 generic.go:356] "Generic (PLEG): container finished" podID="6d001284-d2f0-4e54-af47-26b578dc8596" containerID="26e36941f422b7e1e8334db4d17477f201af1e587f7a2fc975aa655576a89b2f" exitCode=0
Nov 03 08:24:27.768674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.768527    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" event={"ID":"6d001284-d2f0-4e54-af47-26b578dc8596","Type":"ContainerDied","Data":"26e36941f422b7e1e8334db4d17477f201af1e587f7a2fc975aa655576a89b2f"}
Nov 03 08:24:27.768674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.768568    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb" event={"ID":"6d001284-d2f0-4e54-af47-26b578dc8596","Type":"ContainerDied","Data":"be4f70fa90e772cccf917e55d2c3b4e09cd6bd821662eee58fdbc741213bd378"}
Nov 03 08:24:27.768674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.768589    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be4f70fa90e772cccf917e55d2c3b4e09cd6bd821662eee58fdbc741213bd378"
Nov 03 08:24:27.770441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.770353    6430 generic.go:356] "Generic (PLEG): container finished" podID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerID="5db2038d101c0edbba162c3530431c6e87ad83ab1ae34e0ef1127c593a8ef1d2" exitCode=0
Nov 03 08:24:27.770441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.770404    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" event={"ID":"feacf2a1-9660-4dc5-beec-3406a32e24fe","Type":"ContainerDied","Data":"5db2038d101c0edbba162c3530431c6e87ad83ab1ae34e0ef1127c593a8ef1d2"}
Nov 03 08:24:27.787005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.786974    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:24:27.815676 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.815647    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:24:27.862764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.862733    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:24:27.915798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.915768    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"]
Nov 03 08:24:27.937568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.937536    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"]
Nov 03 08:24:27.939087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.938952    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-credential-socket\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.939087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939016    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-socket\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939335    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-socket\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939374    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-data\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939404    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-token\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-token\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939439    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-kube-api-access-h6wpc\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-kube-api-access-h6wpc\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-kube-api-access-8lcqb\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-kube-api-access-8lcqb\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939507    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-podinfo\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939536    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-envoy\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939600    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istiod-ca-cert\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939636    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-certs\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939670    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-socket\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939708    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d001284-d2f0-4e54-af47-26b578dc8596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d001284-d2f0-4e54-af47-26b578dc8596-istio-podinfo\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939737    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-socket" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939755    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d001284-d2f0-4e54-af47-26b578dc8596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d001284-d2f0-4e54-af47-26b578dc8596-istiod-ca-cert\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939857    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feacf2a1-9660-4dc5-beec-3406a32e24fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacf2a1-9660-4dc5-beec-3406a32e24fe-istiod-ca-cert\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939908    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-istio-token\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-istio-token\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939951    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-envoy\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940045    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-certs\") pod \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\" (UID: \"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940094    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-credential-socket\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940129    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-certs\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940180    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-data\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940208    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-kube-api-access-r6cgs\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-kube-api-access-r6cgs\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940283    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-credential-socket\") pod \"6d001284-d2f0-4e54-af47-26b578dc8596\" (UID: \"6d001284-d2f0-4e54-af47-26b578dc8596\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940315    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-podinfo\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940376    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-envoy\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940415    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-data\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940459    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-token\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-token\") pod \"feacf2a1-9660-4dc5-beec-3406a32e24fe\" (UID: \"feacf2a1-9660-4dc5-beec-3406a32e24fe\") "
Nov 03 08:24:27.941499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.941164    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:27.946871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.939949    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-credential-socket" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940166    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-socket" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940347    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-socket" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.940498    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-data" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.943616    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-credential-socket" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.943711    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-certs" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.943728    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-data" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.944038    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istiod-ca-cert" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.944105    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-token" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.944354    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-certs" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.944481    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-credential-socket" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.945103    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-token" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.946999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.946802    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-podinfo" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:27.950815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.949570    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d001284-d2f0-4e54-af47-26b578dc8596-istiod-ca-cert" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:27.950815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.950449    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-envoy" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.950815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.950700    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-data" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.950815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.950789    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-podinfo" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:27.951247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.950983    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-certs" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.951247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.951020    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feacf2a1-9660-4dc5-beec-3406a32e24fe-istiod-ca-cert" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:27.952189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.952163    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-kube-api-access-8lcqb" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "kube-api-access-8lcqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.952309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.952203    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-istio-token" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.952309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.952223    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-kube-api-access-h6wpc" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "kube-api-access-h6wpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.952753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.952731    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-envoy" pod "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" (UID: "d648aabf-a0e9-476b-a8d4-35efc9a9aaa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.953599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.953567    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-envoy" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.954213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.954163    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-kube-api-access-r6cgs" pod "feacf2a1-9660-4dc5-beec-3406a32e24fe" (UID: "feacf2a1-9660-4dc5-beec-3406a32e24fe"). InnerVolumeSpecName "kube-api-access-r6cgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.955322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:27.955299    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d001284-d2f0-4e54-af47-26b578dc8596-istio-podinfo" pod "6d001284-d2f0-4e54-af47-26b578dc8596" (UID: "6d001284-d2f0-4e54-af47-26b578dc8596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:28.027251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.027158    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:28.043263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043216    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043265    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043283    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043298    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d001284-d2f0-4e54-af47-26b578dc8596-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043313    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d001284-d2f0-4e54-af47-26b578dc8596-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043328    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacf2a1-9660-4dc5-beec-3406a32e24fe-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043340    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043353    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043366    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043380    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043393    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043406    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043419    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6cgs\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-kube-api-access-r6cgs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043435    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d001284-d2f0-4e54-af47-26b578dc8596-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043448    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043461    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043475    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043488    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacf2a1-9660-4dc5-beec-3406a32e24fe-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043502    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043516    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043532    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043561    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043574    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6wpc\" (UniqueName: \"kubernetes.io/projected/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-kube-api-access-h6wpc\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043587    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lcqb\" (UniqueName: \"kubernetes.io/projected/6d001284-d2f0-4e54-af47-26b578dc8596-kube-api-access-8lcqb\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043599    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.043963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.043612    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.138084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.136034    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:28.381347 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:28.381249    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:24:28.779111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.779076    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"
Nov 03 08:24:28.779111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.779101    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp" event={"ID":"d648aabf-a0e9-476b-a8d4-35efc9a9aaa6","Type":"ContainerDied","Data":"b9835b764811a4998db4be89a17d4ad6c617115fff17dd230980352dd3c7c59a"}
Nov 03 08:24:28.779371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.779162    6430 scope.go:122] "RemoveContainer" containerID="d1b62c7e3293ff58ec82c11b61d309ef11dd6d5bdf98138ec9c515b3e57835cb"
Nov 03 08:24:28.780848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.780828    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"
Nov 03 08:24:28.780951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.780856    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"
Nov 03 08:24:28.780951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.780852    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n" event={"ID":"feacf2a1-9660-4dc5-beec-3406a32e24fe","Type":"ContainerDied","Data":"44cb11b67735d13248d147e1527937026b80675ca9030a78ca46e4529da452d4"}
Nov 03 08:24:28.790866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.790837    6430 scope.go:122] "RemoveContainer" containerID="5db2038d101c0edbba162c3530431c6e87ad83ab1ae34e0ef1127c593a8ef1d2"
Nov 03 08:24:28.811718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.811665    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"]
Nov 03 08:24:28.815511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.815478    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-64d7n"]
Nov 03 08:24:28.827080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.827031    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"]
Nov 03 08:24:28.831452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.831421    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-gg5tp"]
Nov 03 08:24:28.840494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.840460    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"]
Nov 03 08:24:28.846672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.846639    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-lw9bb"]
Nov 03 08:24:28.877195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:28.877145    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:29.567858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:29.567791    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:30.342202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:30.342168    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:30.618594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:30.618521    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" path="/var/lib/kubelet/pods/6d001284-d2f0-4e54-af47-26b578dc8596/volumes"
Nov 03 08:24:30.619005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:30.618990    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" path="/var/lib/kubelet/pods/d648aabf-a0e9-476b-a8d4-35efc9a9aaa6/volumes"
Nov 03 08:24:30.619480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:30.619466    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" path="/var/lib/kubelet/pods/feacf2a1-9660-4dc5-beec-3406a32e24fe/volumes"
Nov 03 08:24:31.369465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:31.369427    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:32.130322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:32.130278    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:32.359087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:32.353699    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:32.469497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:32.469456    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:32.655410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:32.655373    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:33.061961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:33.061928    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:33.411672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:33.411578    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:33.523352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:33.523320    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:34.040491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:34.040460    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:34.777185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:34.777149    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:35.602006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:35.601970    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:36.459498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:36.459466    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:37.333149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:37.333112    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:38.234054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:38.233999    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:38.834332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:38.834288    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:38.945793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:38.945755    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:39.066180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:39.066143    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:39.102216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:39.102123    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"]
Nov 03 08:24:39.102394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:39.102362    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy" containerID="cri-o://e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb" gracePeriod=30
Nov 03 08:24:39.105948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:39.105919    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"]
Nov 03 08:24:39.106188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:39.106162    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy" containerID="cri-o://227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f" gracePeriod=30
Nov 03 08:24:39.203032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:39.198223    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:39.599505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:39.599465    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:40.139173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:40.139116    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:41.151199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:41.151164    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:41.850183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:41.850147    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:42.641425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:42.641392    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:42.941928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:42.941895    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" containerID="cri-o://a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9" gracePeriod=45
Nov 03 08:24:42.967563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:42.967500    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main" containerID="cri-o://b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298" gracePeriod=45
Nov 03 08:24:43.260475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.260449    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:24:43.264011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.263989    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:24:43.425602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425568    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-home\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-home\") pod \"6ba467c2-f014-4052-8ebc-e769cb311877\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") "
Nov 03 08:24:43.425794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425627    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tls-certs\") pod \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") "
Nov 03 08:24:43.425794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425666    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-model-cache\") pod \"6ba467c2-f014-4052-8ebc-e769cb311877\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") "
Nov 03 08:24:43.425794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425702    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba467c2-f014-4052-8ebc-e769cb311877-kube-api-access-h6q6h\" (UniqueName: \"kubernetes.io/projected/6ba467c2-f014-4052-8ebc-e769cb311877-kube-api-access-h6q6h\") pod \"6ba467c2-f014-4052-8ebc-e769cb311877\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") "
Nov 03 08:24:43.425794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425729    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-dshm\") pod \"6ba467c2-f014-4052-8ebc-e769cb311877\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") "
Nov 03 08:24:43.426024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425831    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-kube-api-access-czb6j\" (UniqueName: \"kubernetes.io/projected/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-kube-api-access-czb6j\") pod \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") "
Nov 03 08:24:43.426024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425898    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-home\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-home\") pod \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") "
Nov 03 08:24:43.426024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425883    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-home" pod "6ba467c2-f014-4052-8ebc-e769cb311877" (UID: "6ba467c2-f014-4052-8ebc-e769cb311877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.426024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425964    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-model-cache" pod "6ba467c2-f014-4052-8ebc-e769cb311877" (UID: "6ba467c2-f014-4052-8ebc-e769cb311877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.426024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.425966    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tmp-dir\") pod \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") "
Nov 03 08:24:43.426327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426084    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-dshm\") pod \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") "
Nov 03 08:24:43.426327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426149    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba467c2-f014-4052-8ebc-e769cb311877-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba467c2-f014-4052-8ebc-e769cb311877-tls-certs\") pod \"6ba467c2-f014-4052-8ebc-e769cb311877\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") "
Nov 03 08:24:43.426327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426218    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-model-cache\") pod \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\" (UID: \"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938\") "
Nov 03 08:24:43.426327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426249    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-tmp-dir\") pod \"6ba467c2-f014-4052-8ebc-e769cb311877\" (UID: \"6ba467c2-f014-4052-8ebc-e769cb311877\") "
Nov 03 08:24:43.426327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426237    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tmp-dir" pod "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" (UID: "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.426327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426304    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-home" pod "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" (UID: "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.426667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426488    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-model-cache" pod "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" (UID: "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.426667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426654    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-tmp-dir" pod "6ba467c2-f014-4052-8ebc-e769cb311877" (UID: "6ba467c2-f014-4052-8ebc-e769cb311877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.427018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.426932    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.427149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.427100    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.427227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.427152    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.427227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.427168    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.427227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.427180    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.427227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.427195    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.428806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.428781    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tls-certs" pod "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" (UID: "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:43.428920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.428816    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba467c2-f014-4052-8ebc-e769cb311877-tls-certs" pod "6ba467c2-f014-4052-8ebc-e769cb311877" (UID: "6ba467c2-f014-4052-8ebc-e769cb311877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:43.428920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.428877    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-kube-api-access-czb6j" pod "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" (UID: "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938"). InnerVolumeSpecName "kube-api-access-czb6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:43.428998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.428934    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba467c2-f014-4052-8ebc-e769cb311877-kube-api-access-h6q6h" pod "6ba467c2-f014-4052-8ebc-e769cb311877" (UID: "6ba467c2-f014-4052-8ebc-e769cb311877"). InnerVolumeSpecName "kube-api-access-h6q6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:43.429037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.428897    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-dshm" pod "6ba467c2-f014-4052-8ebc-e769cb311877" (UID: "6ba467c2-f014-4052-8ebc-e769cb311877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.430266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.430242    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-dshm" pod "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" (UID: "3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:43.507839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.507753    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:43.528758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.528728    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czb6j\" (UniqueName: \"kubernetes.io/projected/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-kube-api-access-czb6j\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.528758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.528759    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.528955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.528769    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba467c2-f014-4052-8ebc-e769cb311877-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.528955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.528778    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.528955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.528787    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6q6h\" (UniqueName: \"kubernetes.io/projected/6ba467c2-f014-4052-8ebc-e769cb311877-kube-api-access-h6q6h\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.528955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.528796    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba467c2-f014-4052-8ebc-e769cb311877-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.932170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.932138    6430 generic.go:356] "Generic (PLEG): container finished" podID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerID="b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298" exitCode=0
Nov 03 08:24:43.932770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.932187    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"
Nov 03 08:24:43.932770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.932217    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" event={"ID":"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938","Type":"ContainerDied","Data":"b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298"}
Nov 03 08:24:43.932770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.932263    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr" event={"ID":"3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938","Type":"ContainerDied","Data":"c0748ceafebfed7a7b0e39880097e2387efa07fb1cd0437c6a93daa5082ae8f6"}
Nov 03 08:24:43.932770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.932288    6430 scope.go:122] "RemoveContainer" containerID="b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298"
Nov 03 08:24:43.933796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.933770    6430 generic.go:356] "Generic (PLEG): container finished" podID="6ba467c2-f014-4052-8ebc-e769cb311877" containerID="a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9" exitCode=0
Nov 03 08:24:43.933884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.933812    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"
Nov 03 08:24:43.933884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.933839    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" event={"ID":"6ba467c2-f014-4052-8ebc-e769cb311877","Type":"ContainerDied","Data":"a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9"}
Nov 03 08:24:43.933884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.933871    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" event={"ID":"6ba467c2-f014-4052-8ebc-e769cb311877","Type":"ContainerDied","Data":"e16effe085ba5ef284d714766965bc5dd7af5e6e4cb0ccac82684477bc8b9bfa"}
Nov 03 08:24:43.944119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.943990    6430 scope.go:122] "RemoveContainer" containerID="b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298"
Nov 03 08:24:43.944413 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:43.944390    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298\": container with ID starting with b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298 not found: ID does not exist" containerID="b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298"
Nov 03 08:24:43.944487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.944429    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298"} err="failed to get container status \"b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298\": rpc error: code = NotFound desc = could not find container \"b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298\": container with ID starting with b967f8503f05ee19775b7f0a84c7413a0d86662a0e33763ddec0528f87443298 not found: ID does not exist"
Nov 03 08:24:43.944487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.944453    6430 scope.go:122] "RemoveContainer" containerID="a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9"
Nov 03 08:24:43.957422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.957389    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"]
Nov 03 08:24:43.964425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.964396    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d"]
Nov 03 08:24:43.974030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.974001    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"]
Nov 03 08:24:43.976325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.976300    6430 scope.go:122] "RemoveContainer" containerID="a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9"
Nov 03 08:24:43.976721 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:43.976700    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9\": container with ID starting with a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9 not found: ID does not exist" containerID="a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9"
Nov 03 08:24:43.976817 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.976734    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9"} err="failed to get container status \"a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9\": rpc error: code = NotFound desc = could not find container \"a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9\": container with ID starting with a9727d2370dbc1c8e4848386ff1f8190c1e2d71aa428b44d77ca667be5f5f7d9 not found: ID does not exist"
Nov 03 08:24:43.978034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:43.978016    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/simulator-w3-3e88b7-kserve-59455cc847-2s8nr"]
Nov 03 08:24:44.245239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.245197    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-w3-3e88b7-kserve-5b4f65f5b9-lgp8d" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:24:44.396424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.396395    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:24:44.413399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.413377    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:24:44.459985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.459896    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545214    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-kube-api-access-slgkk\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-kube-api-access-slgkk\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545278    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-token\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-token\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545337    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-data\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545370    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istiod-ca-cert\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545423    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-podinfo\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545475    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-credential-socket\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545504    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-envoy\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545547    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-envoy\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545577    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-data\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545626    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istiod-ca-cert\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545670    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-socket\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545722    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-podinfo\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545748    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-token\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-token\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545795    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-certs\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.545916    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-credential-socket\") pod \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\" (UID: \"a3df2b48-9c28-4c34-bde0-b7e35e55f41f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.546036    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-kube-api-access-566k6\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-kube-api-access-566k6\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.546089    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-socket\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.546154    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-certs\") pod \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\" (UID: \"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f\") "
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.547318    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istiod-ca-cert" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.547891    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-data" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.548218    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-certs" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.548511    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istiod-ca-cert" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:44.549091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.548719    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-socket" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.551556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.551510    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-credential-socket" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.551870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.551750    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-data" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.551946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.551929    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-credential-socket" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.557107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.557047    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-envoy" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.557339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.557311    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-socket" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.557538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.557516    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-certs" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.558945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.558405    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-envoy" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.558945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.558716    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-kube-api-access-slgkk" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "kube-api-access-slgkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.558945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.558808    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-podinfo" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:44.558945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.558918    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-kube-api-access-566k6" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "kube-api-access-566k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.561517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.560805    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-token" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.562441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.562412    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-token" pod "a3df2b48-9c28-4c34-bde0-b7e35e55f41f" (UID: "a3df2b48-9c28-4c34-bde0-b7e35e55f41f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.569088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.568733    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-podinfo" pod "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" (UID: "ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:44.569819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.569795    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"]
Nov 03 08:24:44.583979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.583932    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"]
Nov 03 08:24:44.597589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.595264    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"]
Nov 03 08:24:44.626006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.625386    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" path="/var/lib/kubelet/pods/3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938/volumes"
Nov 03 08:24:44.626237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.626097    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" path="/var/lib/kubelet/pods/6ba467c2-f014-4052-8ebc-e769cb311877/volumes"
Nov 03 08:24:44.647953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.647918    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.647959    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.647975    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.647989    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648033    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566k6\" (UniqueName: \"kubernetes.io/projected/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-kube-api-access-566k6\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648049    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648080    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648093    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slgkk\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-kube-api-access-slgkk\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648106    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648119    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648132    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648145    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648158    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648172    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648185    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648201    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648213    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.648479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.648226    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3df2b48-9c28-4c34-bde0-b7e35e55f41f-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.686310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.686280    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:44.945862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.945829    6430 generic.go:356] "Generic (PLEG): container finished" podID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerID="e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb" exitCode=0
Nov 03 08:24:44.946342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.945901    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"
Nov 03 08:24:44.946342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.945930    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" event={"ID":"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f","Type":"ContainerDied","Data":"e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb"}
Nov 03 08:24:44.946342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.946001    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr" event={"ID":"ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f","Type":"ContainerDied","Data":"d92013637365235c18322eadb4647bf544503fdd4ca7e2b75904c881d6a95525"}
Nov 03 08:24:44.946342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.946033    6430 scope.go:122] "RemoveContainer" containerID="e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb"
Nov 03 08:24:44.948733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.947901    6430 generic.go:356] "Generic (PLEG): container finished" podID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerID="227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f" exitCode=0
Nov 03 08:24:44.948733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.948100    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"
Nov 03 08:24:44.948733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.948053    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" event={"ID":"a3df2b48-9c28-4c34-bde0-b7e35e55f41f","Type":"ContainerDied","Data":"227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f"}
Nov 03 08:24:44.948733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.948242    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc" event={"ID":"a3df2b48-9c28-4c34-bde0-b7e35e55f41f","Type":"ContainerDied","Data":"6dabcba69c5466c67ba3996af8d4e4fb7e29c6574f1d806bb13120324a9b214e"}
Nov 03 08:24:44.978376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.978353    6430 scope.go:122] "RemoveContainer" containerID="e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb"
Nov 03 08:24:44.978655 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:44.978639    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb\": container with ID starting with e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb not found: ID does not exist" containerID="e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb"
Nov 03 08:24:44.978714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.978664    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb"} err="failed to get container status \"e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb\": rpc error: code = NotFound desc = could not find container \"e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb\": container with ID starting with e78da18ecdde043d6c9bca69191dcdb18688fd9f73f427a0508bd11bc58864bb not found: ID does not exist"
Nov 03 08:24:44.978714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.978680    6430 scope.go:122] "RemoveContainer" containerID="227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f"
Nov 03 08:24:44.988259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.988242    6430 scope.go:122] "RemoveContainer" containerID="227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f"
Nov 03 08:24:44.988507 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:44.988483    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f\": container with ID starting with 227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f not found: ID does not exist" containerID="227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f"
Nov 03 08:24:44.988567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.988514    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f"} err="failed to get container status \"227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f\": rpc error: code = NotFound desc = could not find container \"227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f\": container with ID starting with 227cf33efe11e1b359218c9e5c31cc29117085a5281ff7c82298cbf367b1f84f not found: ID does not exist"
Nov 03 08:24:44.992600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.992577    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"]
Nov 03 08:24:44.996558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:44.996534    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-zwwnr"]
Nov 03 08:24:45.006609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:45.006584    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"]
Nov 03 08:24:45.011217 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:45.011193    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-czvvc"]
Nov 03 08:24:45.561429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:45.561397    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:46.424535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:46.424499    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:46.619909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:46.619877    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" path="/var/lib/kubelet/pods/a3df2b48-9c28-4c34-bde0-b7e35e55f41f/volumes"
Nov 03 08:24:46.620500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:46.620484    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" path="/var/lib/kubelet/pods/ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f/volumes"
Nov 03 08:24:47.205991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:47.205958    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:48.021264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:48.021234    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:48.941592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:48.941561    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:49.471291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.471259    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"]
Nov 03 08:24:49.471941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.471598    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb" podUID="f79c787b-4338-4a84-9433-ead2be648e61" containerName="maas-api" containerID="cri-o://d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663" gracePeriod=30
Nov 03 08:24:49.542695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.542662    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:49.777963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.776977    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:49.907249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.907166    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:49.916310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.916283    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:24:49.983897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.983867    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-maas-api-tls\") pod \"f79c787b-4338-4a84-9433-ead2be648e61\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") "
Nov 03 08:24:49.984106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.984013    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-metrics-certs\") pod \"f79c787b-4338-4a84-9433-ead2be648e61\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") "
Nov 03 08:24:49.984183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.984113    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79c787b-4338-4a84-9433-ead2be648e61-kube-api-access-kwj4h\" (UniqueName: \"kubernetes.io/projected/f79c787b-4338-4a84-9433-ead2be648e61-kube-api-access-kwj4h\") pod \"f79c787b-4338-4a84-9433-ead2be648e61\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") "
Nov 03 08:24:49.984269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.984248    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79c787b-4338-4a84-9433-ead2be648e61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79c787b-4338-4a84-9433-ead2be648e61-openshift-service-ca\") pod \"f79c787b-4338-4a84-9433-ead2be648e61\" (UID: \"f79c787b-4338-4a84-9433-ead2be648e61\") "
Nov 03 08:24:49.984665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.984636    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79c787b-4338-4a84-9433-ead2be648e61-openshift-service-ca" pod "f79c787b-4338-4a84-9433-ead2be648e61" (UID: "f79c787b-4338-4a84-9433-ead2be648e61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:49.984957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.984934    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79c787b-4338-4a84-9433-ead2be648e61-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:49.987351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.987330    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79c787b-4338-4a84-9433-ead2be648e61-kube-api-access-kwj4h" pod "f79c787b-4338-4a84-9433-ead2be648e61" (UID: "f79c787b-4338-4a84-9433-ead2be648e61"). InnerVolumeSpecName "kube-api-access-kwj4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:49.987627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.987610    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-metrics-certs" pod "f79c787b-4338-4a84-9433-ead2be648e61" (UID: "f79c787b-4338-4a84-9433-ead2be648e61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:49.987855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.987839    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-maas-api-tls" pod "f79c787b-4338-4a84-9433-ead2be648e61" (UID: "f79c787b-4338-4a84-9433-ead2be648e61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:49.992969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.992944    6430 generic.go:356] "Generic (PLEG): container finished" podID="f79c787b-4338-4a84-9433-ead2be648e61" containerID="d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663" exitCode=0
Nov 03 08:24:49.993155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.993008    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"
Nov 03 08:24:49.993155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.993028    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb" event={"ID":"f79c787b-4338-4a84-9433-ead2be648e61","Type":"ContainerDied","Data":"d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663"}
Nov 03 08:24:49.993259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.993153    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb" event={"ID":"f79c787b-4338-4a84-9433-ead2be648e61","Type":"ContainerDied","Data":"6b870685802345da480c14c4056a038892f3b85677a5f03594f928d7a8cba742"}
Nov 03 08:24:49.993259 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:49.993187    6430 scope.go:122] "RemoveContainer" containerID="d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663"
Nov 03 08:24:50.016861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.016839    6430 scope.go:122] "RemoveContainer" containerID="d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663"
Nov 03 08:24:50.017184 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:50.017166    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663\": container with ID starting with d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663 not found: ID does not exist" containerID="d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663"
Nov 03 08:24:50.017267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.017194    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663"} err="failed to get container status \"d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663\": rpc error: code = NotFound desc = could not find container \"d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663\": container with ID starting with d99fc5371027f257456b63fb669a38066aa19c0701afbad969182a2e4b5e4663 not found: ID does not exist"
Nov 03 08:24:50.028278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.028214    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:50.029000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.028976    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"]
Nov 03 08:24:50.042600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.042577    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e88b7-86b8f8c49c-k4pxb"]
Nov 03 08:24:50.086653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.086617    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwj4h\" (UniqueName: \"kubernetes.io/projected/f79c787b-4338-4a84-9433-ead2be648e61-kube-api-access-kwj4h\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:50.086653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.086654    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:50.086909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.086675    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79c787b-4338-4a84-9433-ead2be648e61-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:50.146970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.146939    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:50.264266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.264236    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:50.380317 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.378006    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:50.618807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.618774    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79c787b-4338-4a84-9433-ead2be648e61" path="/var/lib/kubelet/pods/f79c787b-4338-4a84-9433-ead2be648e61/volumes"
Nov 03 08:24:50.850621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:50.850587    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:51.538950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:51.538843    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:52.297916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:52.297877    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:53.037330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:53.037292    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:53.874338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:53.874303    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:54.663113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:54.663079    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:55.537665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:55.537629    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:56.324293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:56.324256    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:56.435006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:56.434976    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:56.607002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:56.606903    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"]
Nov 03 08:24:56.607528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:56.607164    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy" containerID="cri-o://b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d" gracePeriod=30
Nov 03 08:24:56.651978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:56.651949    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:57.250622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:57.250578    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:57.886082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:57.886025    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:58.376050 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:24:58.376015    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:24:58.565397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:58.565360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:59.360399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:59.360371    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:24:59.597897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:59.597855    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main" containerID="cri-o://e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5" gracePeriod=45
Nov 03 08:24:59.614973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:59.614935    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main" containerID="cri-o://621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941" gracePeriod=45
Nov 03 08:24:59.640935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:59.640895    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main" containerID="cri-o://b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c" gracePeriod=45
Nov 03 08:24:59.928022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:59.927995    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:24:59.955002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:59.954970    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:24:59.967444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:24:59.967416    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:25:00.057513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057474    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-model-cache\") pod \"2abdfcc3-5815-49a3-8459-6324e5013873\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") "
Nov 03 08:25:00.057705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057570    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tls-certs\") pod \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") "
Nov 03 08:25:00.057705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057605    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-dshm\") pod \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") "
Nov 03 08:25:00.057705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057697    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tmp-dir\") pod \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") "
Nov 03 08:25:00.057882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057732    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-tmp-dir\") pod \"2abdfcc3-5815-49a3-8459-6324e5013873\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") "
Nov 03 08:25:00.057882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057758    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-dshm\") pod \"2abdfcc3-5815-49a3-8459-6324e5013873\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") "
Nov 03 08:25:00.057882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057819    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2abdfcc3-5815-49a3-8459-6324e5013873-tls-certs\" (UniqueName: \"kubernetes.io/secret/2abdfcc3-5815-49a3-8459-6324e5013873-tls-certs\") pod \"2abdfcc3-5815-49a3-8459-6324e5013873\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") "
Nov 03 08:25:00.057882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057849    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba4e9c9-971e-4be9-80db-813f7e1a70ee-kube-api-access-fctfj\" (UniqueName: \"kubernetes.io/projected/bba4e9c9-971e-4be9-80db-813f7e1a70ee-kube-api-access-fctfj\") pod \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") "
Nov 03 08:25:00.058153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057968    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-model-cache\") pod \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") "
Nov 03 08:25:00.058153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.058004    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-home\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-home\") pod \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\" (UID: \"bba4e9c9-971e-4be9-80db-813f7e1a70ee\") "
Nov 03 08:25:00.058153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.058049    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-home\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-home\") pod \"2abdfcc3-5815-49a3-8459-6324e5013873\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") "
Nov 03 08:25:00.058153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.058116    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abdfcc3-5815-49a3-8459-6324e5013873-kube-api-access-gbrft\" (UniqueName: \"kubernetes.io/projected/2abdfcc3-5815-49a3-8459-6324e5013873-kube-api-access-gbrft\") pod \"2abdfcc3-5815-49a3-8459-6324e5013873\" (UID: \"2abdfcc3-5815-49a3-8459-6324e5013873\") "
Nov 03 08:25:00.059347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.057995    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tmp-dir" pod "bba4e9c9-971e-4be9-80db-813f7e1a70ee" (UID: "bba4e9c9-971e-4be9-80db-813f7e1a70ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.059347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.058709    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.059347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.058716    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-model-cache" pod "2abdfcc3-5815-49a3-8459-6324e5013873" (UID: "2abdfcc3-5815-49a3-8459-6324e5013873"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.059347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.059195    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-home" pod "2abdfcc3-5815-49a3-8459-6324e5013873" (UID: "2abdfcc3-5815-49a3-8459-6324e5013873"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.059347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.058932    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-model-cache" pod "bba4e9c9-971e-4be9-80db-813f7e1a70ee" (UID: "bba4e9c9-971e-4be9-80db-813f7e1a70ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.060126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.059972    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-tmp-dir" pod "2abdfcc3-5815-49a3-8459-6324e5013873" (UID: "2abdfcc3-5815-49a3-8459-6324e5013873"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.061944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.061886    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tls-certs" pod "bba4e9c9-971e-4be9-80db-813f7e1a70ee" (UID: "bba4e9c9-971e-4be9-80db-813f7e1a70ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:25:00.061944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.061918    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-dshm" pod "bba4e9c9-971e-4be9-80db-813f7e1a70ee" (UID: "bba4e9c9-971e-4be9-80db-813f7e1a70ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.061944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.061928    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2abdfcc3-5815-49a3-8459-6324e5013873-tls-certs" pod "2abdfcc3-5815-49a3-8459-6324e5013873" (UID: "2abdfcc3-5815-49a3-8459-6324e5013873"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:25:00.062182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.061949    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-home" pod "bba4e9c9-971e-4be9-80db-813f7e1a70ee" (UID: "bba4e9c9-971e-4be9-80db-813f7e1a70ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.062299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.062275    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-dshm" pod "2abdfcc3-5815-49a3-8459-6324e5013873" (UID: "2abdfcc3-5815-49a3-8459-6324e5013873"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.063141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.063122    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba4e9c9-971e-4be9-80db-813f7e1a70ee-kube-api-access-fctfj" pod "bba4e9c9-971e-4be9-80db-813f7e1a70ee" (UID: "bba4e9c9-971e-4be9-80db-813f7e1a70ee"). InnerVolumeSpecName "kube-api-access-fctfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:00.063692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.063669    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abdfcc3-5815-49a3-8459-6324e5013873-kube-api-access-gbrft" pod "2abdfcc3-5815-49a3-8459-6324e5013873" (UID: "2abdfcc3-5815-49a3-8459-6324e5013873"). InnerVolumeSpecName "kube-api-access-gbrft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:00.068363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.068341    6430 generic.go:356] "Generic (PLEG): container finished" podID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerID="621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941" exitCode=0
Nov 03 08:25:00.068481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.068388    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"
Nov 03 08:25:00.068481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.068432    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" event={"ID":"ca64b314-7012-47d5-9e3e-361468b2b02e","Type":"ContainerDied","Data":"621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941"}
Nov 03 08:25:00.068481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.068475    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69" event={"ID":"ca64b314-7012-47d5-9e3e-361468b2b02e","Type":"ContainerDied","Data":"7abcd3fd61948deedb41efa0f1a65301e53553feb7568804c5b2e7271c6733d7"}
Nov 03 08:25:00.068620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.068492    6430 scope.go:122] "RemoveContainer" containerID="621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941"
Nov 03 08:25:00.070257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.070233    6430 generic.go:356] "Generic (PLEG): container finished" podID="2abdfcc3-5815-49a3-8459-6324e5013873" containerID="b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c" exitCode=0
Nov 03 08:25:00.070334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.070273    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"
Nov 03 08:25:00.070334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.070298    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" event={"ID":"2abdfcc3-5815-49a3-8459-6324e5013873","Type":"ContainerDied","Data":"b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c"}
Nov 03 08:25:00.070423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.070340    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454" event={"ID":"2abdfcc3-5815-49a3-8459-6324e5013873","Type":"ContainerDied","Data":"af13c4464bdbd3f1972e5009fa2123c513e7635b1acca4e69f2fd4a5e49610be"}
Nov 03 08:25:00.071864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.071837    6430 generic.go:356] "Generic (PLEG): container finished" podID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerID="e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5" exitCode=0
Nov 03 08:25:00.071976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.071896    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"
Nov 03 08:25:00.071976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.071927    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" event={"ID":"bba4e9c9-971e-4be9-80db-813f7e1a70ee","Type":"ContainerDied","Data":"e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5"}
Nov 03 08:25:00.071976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.071964    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw" event={"ID":"bba4e9c9-971e-4be9-80db-813f7e1a70ee","Type":"ContainerDied","Data":"0278d10531d1db951a2b1d11d6cf30e10b4d4518c549c4425d3e8679b1d1efc0"}
Nov 03 08:25:00.085609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.085588    6430 scope.go:122] "RemoveContainer" containerID="621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941"
Nov 03 08:25:00.086077 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:25:00.086034    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941\": container with ID starting with 621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941 not found: ID does not exist" containerID="621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941"
Nov 03 08:25:00.086166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.086095    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941"} err="failed to get container status \"621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941\": rpc error: code = NotFound desc = could not find container \"621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941\": container with ID starting with 621875bed6cbff7cf1147aad92411cceb78aae1db70b597ad4aae0357afa8941 not found: ID does not exist"
Nov 03 08:25:00.086166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.086115    6430 scope.go:122] "RemoveContainer" containerID="b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c"
Nov 03 08:25:00.092501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.092478    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:00.095775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.095753    6430 scope.go:122] "RemoveContainer" containerID="b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c"
Nov 03 08:25:00.096039 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:25:00.096023    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c\": container with ID starting with b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c not found: ID does not exist" containerID="b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c"
Nov 03 08:25:00.096113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.096047    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c"} err="failed to get container status \"b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c\": rpc error: code = NotFound desc = could not find container \"b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c\": container with ID starting with b96ed0f945528158b79ed118c4ccfb70239405a27b89fe454d6a98dffacef10c not found: ID does not exist"
Nov 03 08:25:00.096113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.096104    6430 scope.go:122] "RemoveContainer" containerID="e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5"
Nov 03 08:25:00.116300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.116116    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"]
Nov 03 08:25:00.136308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.136199    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-2-w6-bd085d-kserve-78bdf9bcdc-lfmlw"]
Nov 03 08:25:00.137675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.137648    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"]
Nov 03 08:25:00.137887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.137869    6430 scope.go:122] "RemoveContainer" containerID="e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5"
Nov 03 08:25:00.138330 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:25:00.138303    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5\": container with ID starting with e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5 not found: ID does not exist" containerID="e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5"
Nov 03 08:25:00.138410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.138341    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5"} err="failed to get container status \"e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5\": rpc error: code = NotFound desc = could not find container \"e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5\": container with ID starting with e4bcfb6af7a95fc689da4b8e2a7ceb3cf4e884ceacffcc385855940bce1346e5 not found: ID does not exist"
Nov 03 08:25:00.142900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.142868    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/simulator-w6-bd085d-kserve-64564f968-56454"]
Nov 03 08:25:00.159367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159278    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-dshm\") pod \"ca64b314-7012-47d5-9e3e-361468b2b02e\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") "
Nov 03 08:25:00.159367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159333    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-model-cache\") pod \"ca64b314-7012-47d5-9e3e-361468b2b02e\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") "
Nov 03 08:25:00.159367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159363    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-home\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-home\") pod \"ca64b314-7012-47d5-9e3e-361468b2b02e\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") "
Nov 03 08:25:00.159685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159592    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca64b314-7012-47d5-9e3e-361468b2b02e-kube-api-access-lkrvz\" (UniqueName: \"kubernetes.io/projected/ca64b314-7012-47d5-9e3e-361468b2b02e-kube-api-access-lkrvz\") pod \"ca64b314-7012-47d5-9e3e-361468b2b02e\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") "
Nov 03 08:25:00.159685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159650    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca64b314-7012-47d5-9e3e-361468b2b02e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca64b314-7012-47d5-9e3e-361468b2b02e-tls-certs\") pod \"ca64b314-7012-47d5-9e3e-361468b2b02e\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") "
Nov 03 08:25:00.159685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159648    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-model-cache" pod "ca64b314-7012-47d5-9e3e-361468b2b02e" (UID: "ca64b314-7012-47d5-9e3e-361468b2b02e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159946    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-tmp-dir\") pod \"ca64b314-7012-47d5-9e3e-361468b2b02e\" (UID: \"ca64b314-7012-47d5-9e3e-361468b2b02e\") "
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.159951    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-home" pod "ca64b314-7012-47d5-9e3e-361468b2b02e" (UID: "ca64b314-7012-47d5-9e3e-361468b2b02e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160179    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-tmp-dir" pod "ca64b314-7012-47d5-9e3e-361468b2b02e" (UID: "ca64b314-7012-47d5-9e3e-361468b2b02e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160494    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160515    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba4e9c9-971e-4be9-80db-813f7e1a70ee-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160527    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160536    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160543    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160551    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160562    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160573    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160585    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2abdfcc3-5815-49a3-8459-6324e5013873-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160599    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctfj\" (UniqueName: \"kubernetes.io/projected/bba4e9c9-971e-4be9-80db-813f7e1a70ee-kube-api-access-fctfj\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160610    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160620    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba4e9c9-971e-4be9-80db-813f7e1a70ee-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160627    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2abdfcc3-5815-49a3-8459-6324e5013873-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.160662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.160635    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrft\" (UniqueName: \"kubernetes.io/projected/2abdfcc3-5815-49a3-8459-6324e5013873-kube-api-access-gbrft\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.162358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.162305    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-dshm" pod "ca64b314-7012-47d5-9e3e-361468b2b02e" (UID: "ca64b314-7012-47d5-9e3e-361468b2b02e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.162441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.162424    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca64b314-7012-47d5-9e3e-361468b2b02e-tls-certs" pod "ca64b314-7012-47d5-9e3e-361468b2b02e" (UID: "ca64b314-7012-47d5-9e3e-361468b2b02e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:25:00.163554 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.163528    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca64b314-7012-47d5-9e3e-361468b2b02e-kube-api-access-lkrvz" pod "ca64b314-7012-47d5-9e3e-361468b2b02e" (UID: "ca64b314-7012-47d5-9e3e-361468b2b02e"). InnerVolumeSpecName "kube-api-access-lkrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:00.261608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.261568    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkrvz\" (UniqueName: \"kubernetes.io/projected/ca64b314-7012-47d5-9e3e-361468b2b02e-kube-api-access-lkrvz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.261608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.261600    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca64b314-7012-47d5-9e3e-361468b2b02e-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.261798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.261615    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca64b314-7012-47d5-9e3e-361468b2b02e-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.387996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.387967    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"]
Nov 03 08:25:00.393108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.393053    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/premium-w6-bd085d-kserve-5d4f747589-dhd69"]
Nov 03 08:25:00.619160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.619133    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" path="/var/lib/kubelet/pods/2abdfcc3-5815-49a3-8459-6324e5013873/volumes"
Nov 03 08:25:00.619611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.619596    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" path="/var/lib/kubelet/pods/bba4e9c9-971e-4be9-80db-813f7e1a70ee/volumes"
Nov 03 08:25:00.620020 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.620006    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" path="/var/lib/kubelet/pods/ca64b314-7012-47d5-9e3e-361468b2b02e/volumes"
Nov 03 08:25:00.780140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:00.780109    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:01.466227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.466195    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:01.889868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.889838    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:25:01.977969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.977933    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-credential-socket\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.977969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.977971    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-certs\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978219 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.977995    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istiod-ca-cert\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978219 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978018    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-token\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-token\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978219 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978041    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-socket\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978219 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978191    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-envoy\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978264    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-data\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978290    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-credential-socket" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:01.978465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978343    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-kube-api-access-h7slj\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-kube-api-access-h7slj\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978388    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-podinfo\") pod \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\" (UID: \"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd\") "
Nov 03 08:25:01.978465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978420    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-socket" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:01.978730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978503    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-data" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:01.978730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978635    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-certs" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:01.978730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978704    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istiod-ca-cert" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:25:01.979018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978881    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:01.979018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978910    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:01.979018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978925    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:01.979018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978938    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:01.979018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.978952    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:01.980680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.980656    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-token" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:01.980826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.980811    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-envoy" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:01.981149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.981127    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-kube-api-access-h7slj" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "kube-api-access-h7slj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:01.981240 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:01.981150    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-podinfo" pod "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" (UID: "709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:25:02.080372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.080336    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.080372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.080373    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7slj\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-kube-api-access-h7slj\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.080589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.080391    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.080589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.080405    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.090430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.090395    6430 generic.go:356] "Generic (PLEG): container finished" podID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerID="b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d" exitCode=0
Nov 03 08:25:02.090587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.090446    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"
Nov 03 08:25:02.090587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.090454    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" event={"ID":"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd","Type":"ContainerDied","Data":"b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d"}
Nov 03 08:25:02.090587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.090511    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv" event={"ID":"709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd","Type":"ContainerDied","Data":"ff33857a796e523d21ce17b89b071845520b9d920b48835b42d385d19665938b"}
Nov 03 08:25:02.090587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.090540    6430 scope.go:122] "RemoveContainer" containerID="b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d"
Nov 03 08:25:02.101664 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.101643    6430 scope.go:122] "RemoveContainer" containerID="b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d"
Nov 03 08:25:02.101961 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:25:02.101944    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d\": container with ID starting with b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d not found: ID does not exist" containerID="b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d"
Nov 03 08:25:02.102024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.101970    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d"} err="failed to get container status \"b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d\": rpc error: code = NotFound desc = could not find container \"b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d\": container with ID starting with b329e11dace31d845f5133e23fc4f71cdf3141ce79c73f3657bc9854fda7b01d not found: ID does not exist"
Nov 03 08:25:02.115210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.115183    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"]
Nov 03 08:25:02.121672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.121646    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ffkmv"]
Nov 03 08:25:02.408374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.408265    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:02.617691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:02.617656    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" path="/var/lib/kubelet/pods/709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd/volumes"
Nov 03 08:25:03.071480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:03.071444    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:03.949805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:03.949773    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:04.741721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:04.741684    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:04.854093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:04.844458    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:05.567629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:05.567592    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:05.766128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:05.766091    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:05.948818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:05.948783    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:06.268563 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:06.268466    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:07.072798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:07.072763    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:07.958452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:07.958411    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:09.068498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:09.068465    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:09.951021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:09.950992    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:10.798755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:10.798711    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:11.006081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:11.006035    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:11.126491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:11.126403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:11.255847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:11.255805    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:11.705550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:11.705509    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:11.937846 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:11.937814    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:12.289096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:12.289049    6430 scope.go:122] "RemoveContainer" containerID="26e36941f422b7e1e8334db4d17477f201af1e587f7a2fc975aa655576a89b2f"
Nov 03 08:25:12.664646 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:12.664552    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:12.853789 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:12.853760    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:13.723195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:13.723154    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:14.296736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:14.296686    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:14.538030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:14.537997    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:15.269832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:15.269798    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:16.100998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:16.100965    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:16.848036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:16.848003    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:17.700636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:17.700604    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:17.823308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:17.823274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:17.958093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:17.958010    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:18.137894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:18.137860    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"]
Nov 03 08:25:18.138395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:18.138360    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy" containerID="cri-o://8864974f72ce13924b4effac8982a8286738963642423b888270e76b4b5c0a4e" gracePeriod=30
Nov 03 08:25:18.179177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:18.179145    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:18.548486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:18.548450    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:19.344620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:19.344580    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:19.904834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:19.904803    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:20.015012 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:20.014966    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:20.126955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:20.126916    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:20.850680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:20.850649    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:21.042878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:21.042832    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:21.146745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:21.146659    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:21.681054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:21.681023    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:22.483518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:22.483482    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:22.598085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:22.594151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:23.258752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.258723    6430 generic.go:356] "Generic (PLEG): container finished" podID="b6d752af-c375-437c-bde9-7886542bf533" containerID="8864974f72ce13924b4effac8982a8286738963642423b888270e76b4b5c0a4e" exitCode=0
Nov 03 08:25:23.258872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.258795    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" event={"ID":"b6d752af-c375-437c-bde9-7886542bf533","Type":"ContainerDied","Data":"8864974f72ce13924b4effac8982a8286738963642423b888270e76b4b5c0a4e"}
Nov 03 08:25:23.411003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.410938    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:25:23.512466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512429    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-certs\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512514    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-socket\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512543    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-credential-socket\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512622    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6d752af-c375-437c-bde9-7886542bf533-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d752af-c375-437c-bde9-7886542bf533-istio-podinfo\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512668    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-istio-token\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-istio-token\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512716    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-kube-api-access-4sgp5\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-kube-api-access-4sgp5\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512745    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-envoy\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512758    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-certs" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.512987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.512883    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-data\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.513502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.513052    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-socket" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.513502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.513159    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d752af-c375-437c-bde9-7886542bf533-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d752af-c375-437c-bde9-7886542bf533-istiod-ca-cert\") pod \"b6d752af-c375-437c-bde9-7886542bf533\" (UID: \"b6d752af-c375-437c-bde9-7886542bf533\") "
Nov 03 08:25:23.513502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.513329    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-credential-socket" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.513928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.513904    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d752af-c375-437c-bde9-7886542bf533-istiod-ca-cert" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:25:23.514178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.514157    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-data" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.514953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.514932    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d752af-c375-437c-bde9-7886542bf533-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.515135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.515119    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.515283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.515269    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.515386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.515372    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.515479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.515467    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.515655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.515631    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6d752af-c375-437c-bde9-7886542bf533-istio-podinfo" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:25:23.516040 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.516014    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-istio-token" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:23.522540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.522506    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-kube-api-access-4sgp5" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "kube-api-access-4sgp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:23.522540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.522519    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-envoy" pod "b6d752af-c375-437c-bde9-7886542bf533" (UID: "b6d752af-c375-437c-bde9-7886542bf533"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.534866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.534838    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:23.616487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.616456    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d752af-c375-437c-bde9-7886542bf533-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.616487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.616486    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.616692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.616496    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sgp5\" (UniqueName: \"kubernetes.io/projected/b6d752af-c375-437c-bde9-7886542bf533-kube-api-access-4sgp5\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.616692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:23.616506    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d752af-c375-437c-bde9-7886542bf533-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:24.284207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:24.284172    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m" event={"ID":"b6d752af-c375-437c-bde9-7886542bf533","Type":"ContainerDied","Data":"ef0034f8820bdc7b6221086ab22a624a8449d4296bd9c3b805715701113acc60"}
Nov 03 08:25:24.284402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:24.284240    6430 scope.go:122] "RemoveContainer" containerID="8864974f72ce13924b4effac8982a8286738963642423b888270e76b4b5c0a4e"
Nov 03 08:25:24.284402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:24.284179    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"
Nov 03 08:25:24.305788 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:24.305758    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"]
Nov 03 08:25:24.309194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:24.309170    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-p2r6m"]
Nov 03 08:25:24.330277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:24.330251    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:24.617728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:24.617654    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d752af-c375-437c-bde9-7886542bf533" path="/var/lib/kubelet/pods/b6d752af-c375-437c-bde9-7886542bf533/volumes"
Nov 03 08:25:25.046537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:25.046505    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:25.842701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:25.842663    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:26.635115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:26.635086    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:27.515131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:27.515101    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:28.152739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:28.152685    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:28.375413 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:25:28.375380    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:25:28.487925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:28.487876    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:28.857549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:28.857444    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:29.547354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:29.547319    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:30.293240 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:30.293197    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:31.038893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:31.038852    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:31.767480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:31.767427    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:32.654309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:32.654278    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:33.609354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:33.609319    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:34.453820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:34.453785    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:35.155713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:35.155684    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:35.955466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:35.955416    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:36.797470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:36.797424    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:37.465969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:37.465939    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:38.170555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:38.170517    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:38.867341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:38.867309    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:39.660360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:39.660316    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:40.341228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:40.341190    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:41.046370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:41.046333    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:41.745350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:41.745306    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:42.441712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:42.441667    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:43.098635 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:43.098598    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:43.835485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:43.835449    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:44.537038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:44.537007    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:45.306161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:45.306117    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:46.096234 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:46.096190    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:46.841929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:46.841877    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:47.543891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:47.543848    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:48.250875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:48.250833    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:49.000607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:49.000561    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:49.828101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:49.828050    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:50.636271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:50.636240    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:51.350759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:51.350726    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:52.123822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:52.123786    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:52.858022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:52.857989    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:53.543231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:53.543183    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:54.354690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:54.354656    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:55.067783 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:55.067748    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:55.734777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:55.734743    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:56.441809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:56.441762    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:57.170323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:57.170289    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:57.906767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:57.906736    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:58.374986 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:25:58.374946    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:25:58.685747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:58.684876    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:25:59.539858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:25:59.539823    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:00.347276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:00.347244    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:01.040813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:01.040781    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:01.706931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:01.706895    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:02.548533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:02.548499    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:03.153154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:03.153117    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:03.867500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:03.867465    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:04.563763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:04.563732    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:05.334387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:05.334345    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:06.038928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:06.038892    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:06.731658 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:06.731618    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:07.442649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:07.442608    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:08.228977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:08.228939    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:09.263871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:09.263841    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:10.160385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:10.160353    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:11.053356 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:11.053323    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:11.808596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:11.808426    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:12.576758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:12.576719    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:13.480996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:13.480957    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:14.312536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:14.312474    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:14.449411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:14.449371    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:15.070857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:15.070807    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:15.853987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:15.853952    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:16.643162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:16.643110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:17.360296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:17.360246    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:18.105242 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:18.105205    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:18.853050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:18.853014    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:19.565302 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:19.565263    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:20.342500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:20.342468    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:21.071418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:21.071384    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:21.779224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:21.779190    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:22.564896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:22.564863    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:23.485234 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:23.485187    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:24.267975 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:24.267939    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:25.044337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:25.044303    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:25.810394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:25.810360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:26.681587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:26.681364    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:27.477688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:27.477631    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:28.256049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:28.256010    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:28.376046 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:26:28.376007    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:26:29.087321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:29.087286    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:29.839732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:29.839697    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:30.571861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:30.571817    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:31.413549 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:31.413515    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:32.217979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:32.217945    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:32.971196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:32.971161    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:33.853038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:33.853009    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:34.556663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:34.556633    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:35.319849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:35.319816    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:36.035625 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:36.035589    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:36.758120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:36.758087    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:37.450817 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:37.450786    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:38.134816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:38.134777    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:38.833088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:38.833026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:39.566089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:39.566016    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:40.442803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:40.442769    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:41.158391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:41.158351    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:41.943756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:41.943720    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:42.699213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:42.699175    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:42.958617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:42.958520    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"]
Nov 03 08:26:43.012541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:43.012509    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:43.741156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:43.740965    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:44.617188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:44.617152    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:45.522986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:45.522955    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:46.344251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:46.344220    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:47.139239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:47.139199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:48.065818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:48.065778    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:48.273355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:48.273322    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"]
Nov 03 08:26:48.345955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:48.345867    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:49.031638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:49.031606    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"]
Nov 03 08:26:49.057444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:49.057414    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:49.197348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:49.197291    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:50.337602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:50.337565    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:51.247232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:51.247202    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:52.046196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:52.046165    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:52.871647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:52.871608    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:53.636427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:53.636399    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:54.456131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:54.456078    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:55.157285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:55.157252    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:55.947684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:55.947653    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:56.975087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:56.975009    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:57.956984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:57.956948    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:57.986152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:57.985020    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng" podUID="2387acc4-0876-413d-9714-aafb39f01ee6" containerName="main" containerID="cri-o://3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b" gracePeriod=45
Nov 03 08:26:58.260770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.260746    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:26:58.352554 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352522    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-home\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-home\") pod \"2387acc4-0876-413d-9714-aafb39f01ee6\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") "
Nov 03 08:26:58.352790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352570    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-tmp-dir\") pod \"2387acc4-0876-413d-9714-aafb39f01ee6\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") "
Nov 03 08:26:58.352790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352626    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-model-cache\") pod \"2387acc4-0876-413d-9714-aafb39f01ee6\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") "
Nov 03 08:26:58.352790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352654    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2387acc4-0876-413d-9714-aafb39f01ee6-kube-api-access-hxq4v\" (UniqueName: \"kubernetes.io/projected/2387acc4-0876-413d-9714-aafb39f01ee6-kube-api-access-hxq4v\") pod \"2387acc4-0876-413d-9714-aafb39f01ee6\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") "
Nov 03 08:26:58.352790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352742    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2387acc4-0876-413d-9714-aafb39f01ee6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2387acc4-0876-413d-9714-aafb39f01ee6-tls-certs\") pod \"2387acc4-0876-413d-9714-aafb39f01ee6\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") "
Nov 03 08:26:58.353049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352829    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-dshm\") pod \"2387acc4-0876-413d-9714-aafb39f01ee6\" (UID: \"2387acc4-0876-413d-9714-aafb39f01ee6\") "
Nov 03 08:26:58.353049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352869    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-tmp-dir" pod "2387acc4-0876-413d-9714-aafb39f01ee6" (UID: "2387acc4-0876-413d-9714-aafb39f01ee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:58.353049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.352876    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-home" pod "2387acc4-0876-413d-9714-aafb39f01ee6" (UID: "2387acc4-0876-413d-9714-aafb39f01ee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:58.353303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.353246    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:58.353303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.353255    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-model-cache" pod "2387acc4-0876-413d-9714-aafb39f01ee6" (UID: "2387acc4-0876-413d-9714-aafb39f01ee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:58.353303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.353271    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:58.355143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.355113    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-dshm" pod "2387acc4-0876-413d-9714-aafb39f01ee6" (UID: "2387acc4-0876-413d-9714-aafb39f01ee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:58.355273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.355163    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2387acc4-0876-413d-9714-aafb39f01ee6-kube-api-access-hxq4v" pod "2387acc4-0876-413d-9714-aafb39f01ee6" (UID: "2387acc4-0876-413d-9714-aafb39f01ee6"). InnerVolumeSpecName "kube-api-access-hxq4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:58.355273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.355183    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2387acc4-0876-413d-9714-aafb39f01ee6-tls-certs" pod "2387acc4-0876-413d-9714-aafb39f01ee6" (UID: "2387acc4-0876-413d-9714-aafb39f01ee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:26:58.378410 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:26:58.378379    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:26:58.453960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.453931    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2387acc4-0876-413d-9714-aafb39f01ee6-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:58.453960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.453957    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:58.453960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.453968    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2387acc4-0876-413d-9714-aafb39f01ee6-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:58.454229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.453978    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxq4v\" (UniqueName: \"kubernetes.io/projected/2387acc4-0876-413d-9714-aafb39f01ee6-kube-api-access-hxq4v\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:58.660303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.660265    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:58.885327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.885231    6430 generic.go:356] "Generic (PLEG): container finished" podID="2387acc4-0876-413d-9714-aafb39f01ee6" containerID="3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b" exitCode=0
Nov 03 08:26:58.885327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.885283    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"
Nov 03 08:26:58.885546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.885324    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng" event={"ID":"2387acc4-0876-413d-9714-aafb39f01ee6","Type":"ContainerDied","Data":"3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b"}
Nov 03 08:26:58.885546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.885370    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng" event={"ID":"2387acc4-0876-413d-9714-aafb39f01ee6","Type":"ContainerDied","Data":"3bfd3d959e39a83b10572419d0a4a68a8901a078f57a87c7a8039f0435070a27"}
Nov 03 08:26:58.885546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.885387    6430 scope.go:122] "RemoveContainer" containerID="3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b"
Nov 03 08:26:58.895411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.895390    6430 scope.go:122] "RemoveContainer" containerID="3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b"
Nov 03 08:26:58.895749 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:26:58.895725    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b\": container with ID starting with 3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b not found: ID does not exist" containerID="3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b"
Nov 03 08:26:58.895828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.895763    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b"} err="failed to get container status \"3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b\": rpc error: code = NotFound desc = could not find container \"3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b\": container with ID starting with 3454163552bb93ec05bf74b264713d59c7aaf098a4c5e030f1b16a4b6c90915b not found: ID does not exist"
Nov 03 08:26:58.900547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.900517    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"]
Nov 03 08:26:58.903737 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:58.903709    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a61a48/auth-test-model-a61a48-kserve-856c756b8f-p6bng"]
Nov 03 08:26:59.333075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:59.330543    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"]
Nov 03 08:26:59.375438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:59.375405    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:59.488991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:59.488935    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:59.731866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:59.731832    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"]
Nov 03 08:26:59.737102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:59.737074    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"]
Nov 03 08:26:59.834765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:59.834732    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:26:59.951260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:26:59.951213    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:00.618314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:00.618280    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2387acc4-0876-413d-9714-aafb39f01ee6" path="/var/lib/kubelet/pods/2387acc4-0876-413d-9714-aafb39f01ee6/volumes"
Nov 03 08:27:00.710696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:00.710658    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:01.551231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:01.551198    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:01.740027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:01.739994    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:01.932169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:01.932130    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:02.054654 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:02.054618    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:02.417408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:02.417336    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:02.967275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:02.967245    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:03.294756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.294669    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q" podUID="42e9de3e-9a77-4f01-980e-72497ced47b4" containerName="main" containerID="cri-o://1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9" gracePeriod=45
Nov 03 08:27:03.565624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.565599    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:27:03.622726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.622699    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e9de3e-9a77-4f01-980e-72497ced47b4-kube-api-access-bjfm6\" (UniqueName: \"kubernetes.io/projected/42e9de3e-9a77-4f01-980e-72497ced47b4-kube-api-access-bjfm6\") pod \"42e9de3e-9a77-4f01-980e-72497ced47b4\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") "
Nov 03 08:27:03.622960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.622781    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-home\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-home\") pod \"42e9de3e-9a77-4f01-980e-72497ced47b4\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") "
Nov 03 08:27:03.622960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.622829    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-tmp-dir\") pod \"42e9de3e-9a77-4f01-980e-72497ced47b4\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") "
Nov 03 08:27:03.622960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.622860    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-dshm\") pod \"42e9de3e-9a77-4f01-980e-72497ced47b4\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") "
Nov 03 08:27:03.622960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.622904    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e9de3e-9a77-4f01-980e-72497ced47b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/42e9de3e-9a77-4f01-980e-72497ced47b4-tls-certs\") pod \"42e9de3e-9a77-4f01-980e-72497ced47b4\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") "
Nov 03 08:27:03.622960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.622945    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-model-cache\") pod \"42e9de3e-9a77-4f01-980e-72497ced47b4\" (UID: \"42e9de3e-9a77-4f01-980e-72497ced47b4\") "
Nov 03 08:27:03.623264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.623127    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-tmp-dir" pod "42e9de3e-9a77-4f01-980e-72497ced47b4" (UID: "42e9de3e-9a77-4f01-980e-72497ced47b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:03.623264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.623145    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-home" pod "42e9de3e-9a77-4f01-980e-72497ced47b4" (UID: "42e9de3e-9a77-4f01-980e-72497ced47b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:03.623366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.623298    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-model-cache" pod "42e9de3e-9a77-4f01-980e-72497ced47b4" (UID: "42e9de3e-9a77-4f01-980e-72497ced47b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:03.623528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.623502    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.623528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.623530    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.623894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.623543    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.625229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.625204    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-dshm" pod "42e9de3e-9a77-4f01-980e-72497ced47b4" (UID: "42e9de3e-9a77-4f01-980e-72497ced47b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:03.625320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.625229    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e9de3e-9a77-4f01-980e-72497ced47b4-kube-api-access-bjfm6" pod "42e9de3e-9a77-4f01-980e-72497ced47b4" (UID: "42e9de3e-9a77-4f01-980e-72497ced47b4"). InnerVolumeSpecName "kube-api-access-bjfm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:03.625383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.625325    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e9de3e-9a77-4f01-980e-72497ced47b4-tls-certs" pod "42e9de3e-9a77-4f01-980e-72497ced47b4" (UID: "42e9de3e-9a77-4f01-980e-72497ced47b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:03.652582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.652550    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:03.724541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.724497    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfm6\" (UniqueName: \"kubernetes.io/projected/42e9de3e-9a77-4f01-980e-72497ced47b4-kube-api-access-bjfm6\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.724541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.724526    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e9de3e-9a77-4f01-980e-72497ced47b4-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.724541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.724539    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e9de3e-9a77-4f01-980e-72497ced47b4-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.808482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.808143    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:03.917750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.917712    6430 generic.go:356] "Generic (PLEG): container finished" podID="42e9de3e-9a77-4f01-980e-72497ced47b4" containerID="1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9" exitCode=0
Nov 03 08:27:03.917934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.917757    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"
Nov 03 08:27:03.917934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.917793    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q" event={"ID":"42e9de3e-9a77-4f01-980e-72497ced47b4","Type":"ContainerDied","Data":"1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9"}
Nov 03 08:27:03.917934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.917837    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q" event={"ID":"42e9de3e-9a77-4f01-980e-72497ced47b4","Type":"ContainerDied","Data":"05674532cc97cc66c250399a308ebf3f236f2affda1644a49a152715c76153e1"}
Nov 03 08:27:03.917934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.917860    6430 scope.go:122] "RemoveContainer" containerID="1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9"
Nov 03 08:27:03.929624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.929599    6430 scope.go:122] "RemoveContainer" containerID="1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9"
Nov 03 08:27:03.929958 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:03.929933    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9\": container with ID starting with 1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9 not found: ID does not exist" containerID="1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9"
Nov 03 08:27:03.930049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.929966    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9"} err="failed to get container status \"1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9\": rpc error: code = NotFound desc = could not find container \"1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9\": container with ID starting with 1896932b3c547d44b1e5fb9373adb839cf0642a84d1c9290edfbe49e995abce9 not found: ID does not exist"
Nov 03 08:27:03.941813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.941782    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"]
Nov 03 08:27:03.944427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:03.944402    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a66d5c/test-model-a66d5c-kserve-6d4956fcd9-6qm5q"]
Nov 03 08:27:04.053288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.053245    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f" podUID="04905656-0d4f-427d-a89f-9eba8e9a17b0" containerName="main" containerID="cri-o://8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775" gracePeriod=45
Nov 03 08:27:04.321782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.321718    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:27:04.432904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.432873    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-tmp-dir\") pod \"04905656-0d4f-427d-a89f-9eba8e9a17b0\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") "
Nov 03 08:27:04.433106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.432974    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-model-cache\") pod \"04905656-0d4f-427d-a89f-9eba8e9a17b0\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") "
Nov 03 08:27:04.433106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433003    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04905656-0d4f-427d-a89f-9eba8e9a17b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/04905656-0d4f-427d-a89f-9eba8e9a17b0-tls-certs\") pod \"04905656-0d4f-427d-a89f-9eba8e9a17b0\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") "
Nov 03 08:27:04.433106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433038    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-home\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-home\") pod \"04905656-0d4f-427d-a89f-9eba8e9a17b0\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") "
Nov 03 08:27:04.433106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433105    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-dshm\") pod \"04905656-0d4f-427d-a89f-9eba8e9a17b0\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") "
Nov 03 08:27:04.433290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433142    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04905656-0d4f-427d-a89f-9eba8e9a17b0-kube-api-access-9s52d\" (UniqueName: \"kubernetes.io/projected/04905656-0d4f-427d-a89f-9eba8e9a17b0-kube-api-access-9s52d\") pod \"04905656-0d4f-427d-a89f-9eba8e9a17b0\" (UID: \"04905656-0d4f-427d-a89f-9eba8e9a17b0\") "
Nov 03 08:27:04.433290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433167    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-tmp-dir" pod "04905656-0d4f-427d-a89f-9eba8e9a17b0" (UID: "04905656-0d4f-427d-a89f-9eba8e9a17b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:04.433405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433308    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-model-cache" pod "04905656-0d4f-427d-a89f-9eba8e9a17b0" (UID: "04905656-0d4f-427d-a89f-9eba8e9a17b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:04.433405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433317    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-home" pod "04905656-0d4f-427d-a89f-9eba8e9a17b0" (UID: "04905656-0d4f-427d-a89f-9eba8e9a17b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:04.433562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433544    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:04.433620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433564    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:04.433620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.433577    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:04.435423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.435397    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-dshm" pod "04905656-0d4f-427d-a89f-9eba8e9a17b0" (UID: "04905656-0d4f-427d-a89f-9eba8e9a17b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:04.435541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.435508    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04905656-0d4f-427d-a89f-9eba8e9a17b0-tls-certs" pod "04905656-0d4f-427d-a89f-9eba8e9a17b0" (UID: "04905656-0d4f-427d-a89f-9eba8e9a17b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:04.435611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.435535    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04905656-0d4f-427d-a89f-9eba8e9a17b0-kube-api-access-9s52d" pod "04905656-0d4f-427d-a89f-9eba8e9a17b0" (UID: "04905656-0d4f-427d-a89f-9eba8e9a17b0"). InnerVolumeSpecName "kube-api-access-9s52d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:04.534566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.534538    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04905656-0d4f-427d-a89f-9eba8e9a17b0-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:04.534566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.534561    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04905656-0d4f-427d-a89f-9eba8e9a17b0-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:04.534715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.534572    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s52d\" (UniqueName: \"kubernetes.io/projected/04905656-0d4f-427d-a89f-9eba8e9a17b0-kube-api-access-9s52d\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:04.602997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.602932    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:04.644363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.644031    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e9de3e-9a77-4f01-980e-72497ced47b4" path="/var/lib/kubelet/pods/42e9de3e-9a77-4f01-980e-72497ced47b4/volumes"
Nov 03 08:27:04.732399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.732367    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" podUID="88feb840-2575-4c86-a9c2-b526ad23c84c" containerName="payload-processing" containerID="cri-o://de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c" gracePeriod=25
Nov 03 08:27:04.737870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.737842    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" podUID="9368968e-6637-402a-998f-8075b5dddfb7" containerName="payload-pre-processing" containerID="cri-o://0b1899b5450039c9bef7334bc3852f086612f51176e5a7b75f8ab87e74e02bf8" gracePeriod=25
Nov 03 08:27:04.925258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.925174    6430 generic.go:356] "Generic (PLEG): container finished" podID="04905656-0d4f-427d-a89f-9eba8e9a17b0" containerID="8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775" exitCode=0
Nov 03 08:27:04.925258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.925215    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"
Nov 03 08:27:04.925495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.925260    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f" event={"ID":"04905656-0d4f-427d-a89f-9eba8e9a17b0","Type":"ContainerDied","Data":"8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775"}
Nov 03 08:27:04.925495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.925309    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f" event={"ID":"04905656-0d4f-427d-a89f-9eba8e9a17b0","Type":"ContainerDied","Data":"4e0e0f42b871a82ebc2c52541fdf21532f824afc0f2464b06fe5eb63b438b5ee"}
Nov 03 08:27:04.925495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.925333    6430 scope.go:122] "RemoveContainer" containerID="8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775"
Nov 03 08:27:04.934596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.934579    6430 scope.go:122] "RemoveContainer" containerID="8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775"
Nov 03 08:27:04.934879 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:04.934857    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775\": container with ID starting with 8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775 not found: ID does not exist" containerID="8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775"
Nov 03 08:27:04.934939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.934888    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775"} err="failed to get container status \"8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775\": rpc error: code = NotFound desc = could not find container \"8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775\": container with ID starting with 8b2abb9f3f98c08f5dd42051f766d4417b57dc5448e6fbb5f1795f380388f775 not found: ID does not exist"
Nov 03 08:27:04.939802 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.939775    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"]
Nov 03 08:27:04.942074 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:04.942044    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d6a8bf/auth-test-model-d6a8bf-kserve-b8c976889-4bd6f"]
Nov 03 08:27:05.417448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:05.417414    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:06.247180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:06.247151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:06.617104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:06.616984    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04905656-0d4f-427d-a89f-9eba8e9a17b0" path="/var/lib/kubelet/pods/04905656-0d4f-427d-a89f-9eba8e9a17b0/volumes"
Nov 03 08:27:06.962637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:06.962603    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:07.712339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:07.712304    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:07.900083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:07.892545    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" podUID="88feb840-2575-4c86-a9c2-b526ad23c84c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:07.900886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:07.900823    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" podUID="9368968e-6637-402a-998f-8075b5dddfb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:08.531667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:08.531635    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:09.233149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:09.233110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:09.347458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:09.347423    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:09.467039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:09.466998    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:09.513103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:09.512931    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"]
Nov 03 08:27:09.513358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:09.513195    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy" containerID="cri-o://13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e" gracePeriod=30
Nov 03 08:27:09.582290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:09.582257    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:09.980013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:09.979888    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:10.091705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:10.091675    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:10.543158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:10.543127    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:11.443493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:11.443434    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:12.333383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:12.333349    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:13.008510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:13.008473    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:13.783498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:13.783430    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:14.364082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.363996    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg" podUID="1b7c09ad-494e-412e-9d98-093a3e5df344" containerName="main" containerID="cri-o://139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc" gracePeriod=45
Nov 03 08:27:14.400937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.400900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:14.651142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.651118    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:27:14.723087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.720371    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:14.772028 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.771999    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7c09ad-494e-412e-9d98-093a3e5df344-kube-api-access-qqq6v\" (UniqueName: \"kubernetes.io/projected/1b7c09ad-494e-412e-9d98-093a3e5df344-kube-api-access-qqq6v\") pod \"1b7c09ad-494e-412e-9d98-093a3e5df344\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") "
Nov 03 08:27:14.772192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772098    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-home\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-home\") pod \"1b7c09ad-494e-412e-9d98-093a3e5df344\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") "
Nov 03 08:27:14.772192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772155    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-model-cache\") pod \"1b7c09ad-494e-412e-9d98-093a3e5df344\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") "
Nov 03 08:27:14.772304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772257    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-dshm\") pod \"1b7c09ad-494e-412e-9d98-093a3e5df344\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") "
Nov 03 08:27:14.772366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772303    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7c09ad-494e-412e-9d98-093a3e5df344-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7c09ad-494e-412e-9d98-093a3e5df344-tls-certs\") pod \"1b7c09ad-494e-412e-9d98-093a3e5df344\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") "
Nov 03 08:27:14.772422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772403    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-tmp-dir\") pod \"1b7c09ad-494e-412e-9d98-093a3e5df344\" (UID: \"1b7c09ad-494e-412e-9d98-093a3e5df344\") "
Nov 03 08:27:14.772476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772422    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-home" pod "1b7c09ad-494e-412e-9d98-093a3e5df344" (UID: "1b7c09ad-494e-412e-9d98-093a3e5df344"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.772476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772431    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-model-cache" pod "1b7c09ad-494e-412e-9d98-093a3e5df344" (UID: "1b7c09ad-494e-412e-9d98-093a3e5df344"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.772710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772687    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-tmp-dir" pod "1b7c09ad-494e-412e-9d98-093a3e5df344" (UID: "1b7c09ad-494e-412e-9d98-093a3e5df344"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.772827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772816    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.772891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772834    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.772891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.772848    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.775146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.775118    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-dshm" pod "1b7c09ad-494e-412e-9d98-093a3e5df344" (UID: "1b7c09ad-494e-412e-9d98-093a3e5df344"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.775241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.775177    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7c09ad-494e-412e-9d98-093a3e5df344-tls-certs" pod "1b7c09ad-494e-412e-9d98-093a3e5df344" (UID: "1b7c09ad-494e-412e-9d98-093a3e5df344"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:14.775241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.775184    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7c09ad-494e-412e-9d98-093a3e5df344-kube-api-access-qqq6v" pod "1b7c09ad-494e-412e-9d98-093a3e5df344" (UID: "1b7c09ad-494e-412e-9d98-093a3e5df344"). InnerVolumeSpecName "kube-api-access-qqq6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:14.813795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.813775    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:27:14.873565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873481    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-token\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.873565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873536    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-credential-socket\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.873776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873587    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-kube-api-access-grqnx\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-kube-api-access-grqnx\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.873776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873613    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-envoy\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.873776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873657    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-podinfo\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.873776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873713    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istiod-ca-cert\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.873776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873762    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-certs\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.874046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873851    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-credential-socket" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.874046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873861    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-data\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.874046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.873920    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-socket\") pod \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\" (UID: \"8d8e8ea0-340a-4aef-8b26-ee5b79d47394\") "
Nov 03 08:27:14.874249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874096    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-data" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.874249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874225    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-certs" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.874424 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874390    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-socket" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.874527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874503    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istiod-ca-cert" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874644    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7c09ad-494e-412e-9d98-093a3e5df344-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874673    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874690    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874706    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874722    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq6v\" (UniqueName: \"kubernetes.io/projected/1b7c09ad-494e-412e-9d98-093a3e5df344-kube-api-access-qqq6v\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874737    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874754    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.874697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.874767    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7c09ad-494e-412e-9d98-093a3e5df344-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.876002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.875960    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-token" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:14.876002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.875974    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-envoy" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:14.876461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.876442    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-kube-api-access-grqnx" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "kube-api-access-grqnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:14.876589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.876576    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-podinfo" pod "8d8e8ea0-340a-4aef-8b26-ee5b79d47394" (UID: "8d8e8ea0-340a-4aef-8b26-ee5b79d47394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:27:14.975395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.975365    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.975395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.975392    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqnx\" (UniqueName: \"kubernetes.io/projected/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-kube-api-access-grqnx\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.975541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.975410    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.975541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.975422    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8e8ea0-340a-4aef-8b26-ee5b79d47394-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:14.988086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.988048    6430 generic.go:356] "Generic (PLEG): container finished" podID="1b7c09ad-494e-412e-9d98-093a3e5df344" containerID="139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc" exitCode=0
Nov 03 08:27:14.988195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.988086    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg" event={"ID":"1b7c09ad-494e-412e-9d98-093a3e5df344","Type":"ContainerDied","Data":"139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc"}
Nov 03 08:27:14.988195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.988125    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"
Nov 03 08:27:14.988195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.988135    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg" event={"ID":"1b7c09ad-494e-412e-9d98-093a3e5df344","Type":"ContainerDied","Data":"6dfee1425e508143068d9bb81b590429f2cf829a865abc0dbb7b57ab9a24a0c9"}
Nov 03 08:27:14.988195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.988160    6430 scope.go:122] "RemoveContainer" containerID="139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc"
Nov 03 08:27:14.989966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.989941    6430 generic.go:356] "Generic (PLEG): container finished" podID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerID="13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e" exitCode=0
Nov 03 08:27:14.990072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.989997    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"
Nov 03 08:27:14.990072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.990017    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" event={"ID":"8d8e8ea0-340a-4aef-8b26-ee5b79d47394","Type":"ContainerDied","Data":"13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e"}
Nov 03 08:27:14.990181 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.990055    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr" event={"ID":"8d8e8ea0-340a-4aef-8b26-ee5b79d47394","Type":"ContainerDied","Data":"9b12a2d58c717ab63bafdffefda22f600e904f14c9ec11f0db20801d759391df"}
Nov 03 08:27:14.997312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.997295    6430 scope.go:122] "RemoveContainer" containerID="139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc"
Nov 03 08:27:14.997570 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:14.997546    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc\": container with ID starting with 139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc not found: ID does not exist" containerID="139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc"
Nov 03 08:27:14.997684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.997577    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc"} err="failed to get container status \"139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc\": rpc error: code = NotFound desc = could not find container \"139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc\": container with ID starting with 139b7a7cca398934d4cdbe5aa8a430b77833a76ead2ad4f73a911e2180bdd9fc not found: ID does not exist"
Nov 03 08:27:14.997684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:14.997596    6430 scope.go:122] "RemoveContainer" containerID="13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e"
Nov 03 08:27:15.006903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.006884    6430 scope.go:122] "RemoveContainer" containerID="91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce"
Nov 03 08:27:15.015515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.015499    6430 scope.go:122] "RemoveContainer" containerID="13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e"
Nov 03 08:27:15.015808 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:15.015790    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e\": container with ID starting with 13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e not found: ID does not exist" containerID="13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e"
Nov 03 08:27:15.015861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.015816    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e"} err="failed to get container status \"13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e\": rpc error: code = NotFound desc = could not find container \"13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e\": container with ID starting with 13d74cba54552e5b5c334540a9a112c117f5c1c81a6222f7b9b3ac88be30153e not found: ID does not exist"
Nov 03 08:27:15.015861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.015833    6430 scope.go:122] "RemoveContainer" containerID="91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce"
Nov 03 08:27:15.016049 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:15.016029    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce\": container with ID starting with 91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce not found: ID does not exist" containerID="91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce"
Nov 03 08:27:15.016161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.016081    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce"} err="failed to get container status \"91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce\": rpc error: code = NotFound desc = could not find container \"91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce\": container with ID starting with 91db7ab29fd3fafcf323ff6713b97ce0eeaaeb9582336880ebb91559627bdbce not found: ID does not exist"
Nov 03 08:27:15.038913 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.038894    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"]
Nov 03 08:27:15.043605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.043577    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a61a48-openshift-default-7fb6ff8ff9-wjqhr"]
Nov 03 08:27:15.072605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.072581    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"]
Nov 03 08:27:15.083088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.083049    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-109070/test-model-109070-kserve-cc466b949-5v2wg"]
Nov 03 08:27:15.573016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:15.572985    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:16.462910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:16.462869    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:16.617166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:16.617137    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7c09ad-494e-412e-9d98-093a3e5df344" path="/var/lib/kubelet/pods/1b7c09ad-494e-412e-9d98-093a3e5df344/volumes"
Nov 03 08:27:16.617656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:16.617640    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" path="/var/lib/kubelet/pods/8d8e8ea0-340a-4aef-8b26-ee5b79d47394/volumes"
Nov 03 08:27:17.287480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:17.287438    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:17.853588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:17.853555    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"]
Nov 03 08:27:17.854104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:17.853857    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb" podUID="94702209-a919-4445-9577-cd5739032003" containerName="maas-api" containerID="cri-o://bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d" gracePeriod=30
Nov 03 08:27:17.893102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:17.892384    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" podUID="88feb840-2575-4c86-a9c2-b526ad23c84c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:17.893102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:17.892544    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" podUID="9368968e-6637-402a-998f-8075b5dddfb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:17.925722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:17.925688    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:18.086487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.086407    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:18.181994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.181933    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb" podUID="94702209-a919-4445-9577-cd5739032003" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.153:8443/health\": dial tcp 10.133.0.153:8443: connect: connection refused"
Nov 03 08:27:18.198771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.198742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:18.318114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.318053    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:18.337775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.337750    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:27:18.427965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.427882    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94702209-a919-4445-9577-cd5739032003-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94702209-a919-4445-9577-cd5739032003-openshift-service-ca\") pod \"94702209-a919-4445-9577-cd5739032003\" (UID: \"94702209-a919-4445-9577-cd5739032003\") "
Nov 03 08:27:18.428155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.427963    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-metrics-certs\") pod \"94702209-a919-4445-9577-cd5739032003\" (UID: \"94702209-a919-4445-9577-cd5739032003\") "
Nov 03 08:27:18.428155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.428005    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-maas-api-tls\") pod \"94702209-a919-4445-9577-cd5739032003\" (UID: \"94702209-a919-4445-9577-cd5739032003\") "
Nov 03 08:27:18.428155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.428028    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94702209-a919-4445-9577-cd5739032003-kube-api-access-7p5rn\" (UniqueName: \"kubernetes.io/projected/94702209-a919-4445-9577-cd5739032003-kube-api-access-7p5rn\") pod \"94702209-a919-4445-9577-cd5739032003\" (UID: \"94702209-a919-4445-9577-cd5739032003\") "
Nov 03 08:27:18.428365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.428338    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94702209-a919-4445-9577-cd5739032003-openshift-service-ca" pod "94702209-a919-4445-9577-cd5739032003" (UID: "94702209-a919-4445-9577-cd5739032003"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:18.430416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.430389    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-metrics-certs" pod "94702209-a919-4445-9577-cd5739032003" (UID: "94702209-a919-4445-9577-cd5739032003"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:18.430416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.430403    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-maas-api-tls" pod "94702209-a919-4445-9577-cd5739032003" (UID: "94702209-a919-4445-9577-cd5739032003"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:18.430583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.430400    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94702209-a919-4445-9577-cd5739032003-kube-api-access-7p5rn" pod "94702209-a919-4445-9577-cd5739032003" (UID: "94702209-a919-4445-9577-cd5739032003"). InnerVolumeSpecName "kube-api-access-7p5rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:18.529298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.529264    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:18.529298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.529291    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94702209-a919-4445-9577-cd5739032003-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:18.529298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.529300    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5rn\" (UniqueName: \"kubernetes.io/projected/94702209-a919-4445-9577-cd5739032003-kube-api-access-7p5rn\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:18.529550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.529313    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94702209-a919-4445-9577-cd5739032003-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:18.593446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.593411    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:18.971955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:18.971920    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:19.020267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.020232    6430 generic.go:356] "Generic (PLEG): container finished" podID="94702209-a919-4445-9577-cd5739032003" containerID="bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d" exitCode=0
Nov 03 08:27:19.020471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.020284    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"
Nov 03 08:27:19.020471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.020319    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb" event={"ID":"94702209-a919-4445-9577-cd5739032003","Type":"ContainerDied","Data":"bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d"}
Nov 03 08:27:19.020471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.020386    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb" event={"ID":"94702209-a919-4445-9577-cd5739032003","Type":"ContainerDied","Data":"f289e25e71613f336aa8f27ec137a357b9ec30c3135f6bf06bb1c91e32517ab6"}
Nov 03 08:27:19.020471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.020413    6430 scope.go:122] "RemoveContainer" containerID="bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d"
Nov 03 08:27:19.029832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.029815    6430 scope.go:122] "RemoveContainer" containerID="bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d"
Nov 03 08:27:19.030133 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:19.030110    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d\": container with ID starting with bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d not found: ID does not exist" containerID="bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d"
Nov 03 08:27:19.030231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.030138    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d"} err="failed to get container status \"bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d\": rpc error: code = NotFound desc = could not find container \"bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d\": container with ID starting with bf4ccc83debc4195802e6e6c7ad352875c7758e9f018e8315038654a7aca078d not found: ID does not exist"
Nov 03 08:27:19.036895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.036873    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"]
Nov 03 08:27:19.040518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.040500    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a66d5c-f57cdd4fc-bjsfb"]
Nov 03 08:27:19.664665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:19.664625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:20.152638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:20.152603    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"]
Nov 03 08:27:20.250075 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:20.250028    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:20.355468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:20.355438    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:20.456743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:20.456714    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:20.618785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:20.618756    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94702209-a919-4445-9577-cd5739032003" path="/var/lib/kubelet/pods/94702209-a919-4445-9577-cd5739032003/volumes"
Nov 03 08:27:21.210815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:21.210782    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:21.946442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:21.946407    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:22.647953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:22.647920    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:23.365675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:23.365640    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:24.220324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:24.220270    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:24.906754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:24.906717    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:25.018278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.018252    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:27:25.104083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.099383    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:25.104083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.102139    6430 generic.go:356] "Generic (PLEG): container finished" podID="9368968e-6637-402a-998f-8075b5dddfb7" containerID="0b1899b5450039c9bef7334bc3852f086612f51176e5a7b75f8ab87e74e02bf8" exitCode=0
Nov 03 08:27:25.104083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.102399    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" event={"ID":"9368968e-6637-402a-998f-8075b5dddfb7","Type":"ContainerDied","Data":"0b1899b5450039c9bef7334bc3852f086612f51176e5a7b75f8ab87e74e02bf8"}
Nov 03 08:27:25.104083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.103948    6430 generic.go:356] "Generic (PLEG): container finished" podID="88feb840-2575-4c86-a9c2-b526ad23c84c" containerID="de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c" exitCode=0
Nov 03 08:27:25.104083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.103984    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" event={"ID":"88feb840-2575-4c86-a9c2-b526ad23c84c","Type":"ContainerDied","Data":"de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c"}
Nov 03 08:27:25.104083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.104023    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5" event={"ID":"88feb840-2575-4c86-a9c2-b526ad23c84c","Type":"ContainerDied","Data":"b015733f24707da67b86e1e90df96eb21925216f3fc41ac3a790a19691a647d8"}
Nov 03 08:27:25.104083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.104054    6430 scope.go:122] "RemoveContainer" containerID="de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c"
Nov 03 08:27:25.104632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.104204    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"
Nov 03 08:27:25.116506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.116474    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88feb840-2575-4c86-a9c2-b526ad23c84c-kube-api-access-2fvxx\" (UniqueName: \"kubernetes.io/projected/88feb840-2575-4c86-a9c2-b526ad23c84c-kube-api-access-2fvxx\") pod \"88feb840-2575-4c86-a9c2-b526ad23c84c\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") "
Nov 03 08:27:25.116711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.116648    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88feb840-2575-4c86-a9c2-b526ad23c84c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88feb840-2575-4c86-a9c2-b526ad23c84c-plugins-config-volume\") pod \"88feb840-2575-4c86-a9c2-b526ad23c84c\" (UID: \"88feb840-2575-4c86-a9c2-b526ad23c84c\") "
Nov 03 08:27:25.125619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.119658    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88feb840-2575-4c86-a9c2-b526ad23c84c-plugins-config-volume" pod "88feb840-2575-4c86-a9c2-b526ad23c84c" (UID: "88feb840-2575-4c86-a9c2-b526ad23c84c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:25.129237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.127373    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:27:25.135516 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.134946    6430 scope.go:122] "RemoveContainer" containerID="de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c"
Nov 03 08:27:25.136488 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:25.136428    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c\": container with ID starting with de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c not found: ID does not exist" containerID="de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c"
Nov 03 08:27:25.136638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.136489    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c"} err="failed to get container status \"de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c\": rpc error: code = NotFound desc = could not find container \"de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c\": container with ID starting with de44556a315175dcc389440083f489442c60a172ed58de185287dd77b785608c not found: ID does not exist"
Nov 03 08:27:25.139914 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.139825    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88feb840-2575-4c86-a9c2-b526ad23c84c-kube-api-access-2fvxx" pod "88feb840-2575-4c86-a9c2-b526ad23c84c" (UID: "88feb840-2575-4c86-a9c2-b526ad23c84c"). InnerVolumeSpecName "kube-api-access-2fvxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:25.152947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.152911    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" podUID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" containerName="payload-processing" containerID="cri-o://267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033" gracePeriod=25
Nov 03 08:27:25.217967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.217935    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9368968e-6637-402a-998f-8075b5dddfb7-kube-api-access-2hjdw\" (UniqueName: \"kubernetes.io/projected/9368968e-6637-402a-998f-8075b5dddfb7-kube-api-access-2hjdw\") pod \"9368968e-6637-402a-998f-8075b5dddfb7\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") "
Nov 03 08:27:25.218188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.217978    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9368968e-6637-402a-998f-8075b5dddfb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368968e-6637-402a-998f-8075b5dddfb7-plugins-config-volume\") pod \"9368968e-6637-402a-998f-8075b5dddfb7\" (UID: \"9368968e-6637-402a-998f-8075b5dddfb7\") "
Nov 03 08:27:25.218296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.218281    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvxx\" (UniqueName: \"kubernetes.io/projected/88feb840-2575-4c86-a9c2-b526ad23c84c-kube-api-access-2fvxx\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:25.218337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.218301    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88feb840-2575-4c86-a9c2-b526ad23c84c-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:25.218429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.218401    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9368968e-6637-402a-998f-8075b5dddfb7-plugins-config-volume" pod "9368968e-6637-402a-998f-8075b5dddfb7" (UID: "9368968e-6637-402a-998f-8075b5dddfb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:25.220293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.220275    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9368968e-6637-402a-998f-8075b5dddfb7-kube-api-access-2hjdw" pod "9368968e-6637-402a-998f-8075b5dddfb7" (UID: "9368968e-6637-402a-998f-8075b5dddfb7"). InnerVolumeSpecName "kube-api-access-2hjdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:25.262623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.262547    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:25.318964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.318926    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hjdw\" (UniqueName: \"kubernetes.io/projected/9368968e-6637-402a-998f-8075b5dddfb7-kube-api-access-2hjdw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:25.318964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.318961    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368968e-6637-402a-998f-8075b5dddfb7-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:25.400712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.400682    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:25.427164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.427138    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"]
Nov 03 08:27:25.429916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.429889    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a61a48-75f98cf85-ns4m5"]
Nov 03 08:27:25.711887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.711856    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:25.834015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:25.833204    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:26.103105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.103005    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:26.112009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.111969    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"
Nov 03 08:27:26.112182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.111983    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc" event={"ID":"9368968e-6637-402a-998f-8075b5dddfb7","Type":"ContainerDied","Data":"e8232872f15f36b9831f29e583c2ea0d3094868164b77c381f9e8d1af3d2c827"}
Nov 03 08:27:26.112182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.112135    6430 scope.go:122] "RemoveContainer" containerID="0b1899b5450039c9bef7334bc3852f086612f51176e5a7b75f8ab87e74e02bf8"
Nov 03 08:27:26.135348 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.135321    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"]
Nov 03 08:27:26.138641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.138618    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a61a48-5cd584c59-gkqcc"]
Nov 03 08:27:26.218890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.218856    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:26.616342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.616305    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88feb840-2575-4c86-a9c2-b526ad23c84c" path="/var/lib/kubelet/pods/88feb840-2575-4c86-a9c2-b526ad23c84c/volumes"
Nov 03 08:27:26.616704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:26.616691    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9368968e-6637-402a-998f-8075b5dddfb7" path="/var/lib/kubelet/pods/9368968e-6637-402a-998f-8075b5dddfb7/volumes"
Nov 03 08:27:27.155355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:27.155325    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:27.854146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:27.854110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:28.375673 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:28.375633    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:27:28.589670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:28.589639    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:28.631465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:28.631378    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" podUID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:29.394665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:29.394628    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:30.306773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:30.306740    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:31.131820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:31.131792    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:32.121090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:32.121025    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:32.324420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:32.324364    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:32.442850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:32.442820    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:32.655319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:32.655288    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:33.044978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:33.044910    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:33.153982 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:33.153934    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:33.629613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:33.629576    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:33.738477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:33.738446    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:34.035321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:34.035290    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:34.836294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:34.836261    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:35.538043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:35.538013    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:35.985194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:35.985155    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"]
Nov 03 08:27:36.055392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:36.055347    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:36.454953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:36.454919    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:36.746993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:36.746920    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"]
Nov 03 08:27:36.752290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:36.752266    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"]
Nov 03 08:27:36.846205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:36.846175    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:36.962320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:36.962288    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:37.450683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:37.450652    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:38.175851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:38.175822    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:38.293798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:38.293764    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:38.631622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:38.631575    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" podUID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:39.159518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:39.159487    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:40.020517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:40.020487    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:40.750730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:40.750698    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:41.446923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:41.446864    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:41.748195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:41.748110    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" podUID="2d61781c-fa71-42ef-8ada-4350c32675bd" containerName="payload-processing" containerID="cri-o://6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da" gracePeriod=25
Nov 03 08:27:41.752679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:41.752643    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" podUID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" containerName="payload-pre-processing" containerID="cri-o://26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e" gracePeriod=25
Nov 03 08:27:42.156150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:42.156010    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:42.962514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:42.962484    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:43.471727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.471694    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:43.587458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.587422    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"]
Nov 03 08:27:43.587729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.587702    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5" podUID="51912a37-547f-4db7-b5e9-a56b62cfa3bf" containerName="maas-api" containerID="cri-o://8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233" gracePeriod=30
Nov 03 08:27:43.597484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.597454    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:43.709716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.709682    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:43.841482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.841363    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:43.928269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.928239    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:27:43.958714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:43.958611    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:44.042860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.042769    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-maas-api-tls\") pod \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") "
Nov 03 08:27:44.043384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.042892    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-metrics-certs\") pod \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") "
Nov 03 08:27:44.043384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.042971    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51912a37-547f-4db7-b5e9-a56b62cfa3bf-kube-api-access-t6hbc\" (UniqueName: \"kubernetes.io/projected/51912a37-547f-4db7-b5e9-a56b62cfa3bf-kube-api-access-t6hbc\") pod \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") "
Nov 03 08:27:44.043384 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.043035    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51912a37-547f-4db7-b5e9-a56b62cfa3bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51912a37-547f-4db7-b5e9-a56b62cfa3bf-openshift-service-ca\") pod \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\" (UID: \"51912a37-547f-4db7-b5e9-a56b62cfa3bf\") "
Nov 03 08:27:44.043526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.043489    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51912a37-547f-4db7-b5e9-a56b62cfa3bf-openshift-service-ca" pod "51912a37-547f-4db7-b5e9-a56b62cfa3bf" (UID: "51912a37-547f-4db7-b5e9-a56b62cfa3bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:44.045140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.045120    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-maas-api-tls" pod "51912a37-547f-4db7-b5e9-a56b62cfa3bf" (UID: "51912a37-547f-4db7-b5e9-a56b62cfa3bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:44.045679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.045658    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51912a37-547f-4db7-b5e9-a56b62cfa3bf-kube-api-access-t6hbc" pod "51912a37-547f-4db7-b5e9-a56b62cfa3bf" (UID: "51912a37-547f-4db7-b5e9-a56b62cfa3bf"). InnerVolumeSpecName "kube-api-access-t6hbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:44.045679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.045660    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-metrics-certs" pod "51912a37-547f-4db7-b5e9-a56b62cfa3bf" (UID: "51912a37-547f-4db7-b5e9-a56b62cfa3bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:44.144097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.144019    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51912a37-547f-4db7-b5e9-a56b62cfa3bf-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:44.144097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.144097    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:44.144335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.144114    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51912a37-547f-4db7-b5e9-a56b62cfa3bf-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:44.144335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.144127    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6hbc\" (UniqueName: \"kubernetes.io/projected/51912a37-547f-4db7-b5e9-a56b62cfa3bf-kube-api-access-t6hbc\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:44.185688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.185655    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:44.233834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.233792    6430 generic.go:356] "Generic (PLEG): container finished" podID="51912a37-547f-4db7-b5e9-a56b62cfa3bf" containerID="8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233" exitCode=0
Nov 03 08:27:44.234011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.233845    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"
Nov 03 08:27:44.234011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.233895    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5" event={"ID":"51912a37-547f-4db7-b5e9-a56b62cfa3bf","Type":"ContainerDied","Data":"8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233"}
Nov 03 08:27:44.234011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.233961    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5" event={"ID":"51912a37-547f-4db7-b5e9-a56b62cfa3bf","Type":"ContainerDied","Data":"41d1246e39e7bc4fdc0ed1422c6e95134efb53edd5881d7595402f753955a679"}
Nov 03 08:27:44.234011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.233993    6430 scope.go:122] "RemoveContainer" containerID="8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233"
Nov 03 08:27:44.245152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.245130    6430 scope.go:122] "RemoveContainer" containerID="8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233"
Nov 03 08:27:44.245452 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:44.245434    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233\": container with ID starting with 8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233 not found: ID does not exist" containerID="8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233"
Nov 03 08:27:44.245504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.245461    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233"} err="failed to get container status \"8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233\": rpc error: code = NotFound desc = could not find container \"8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233\": container with ID starting with 8b5abdb7c5ef4175dddd9c9d0112f9fae9ad1de6a619e47c66ce9027445fe233 not found: ID does not exist"
Nov 03 08:27:44.258202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.258170    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"]
Nov 03 08:27:44.261652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.261625    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-109070-5bf789b96-xrbj5"]
Nov 03 08:27:44.433231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.433201    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:44.618135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.618105    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51912a37-547f-4db7-b5e9-a56b62cfa3bf" path="/var/lib/kubelet/pods/51912a37-547f-4db7-b5e9-a56b62cfa3bf/volumes"
Nov 03 08:27:44.960429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:44.960399    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:45.412854 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.412826    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:27:45.458107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.458048    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-kube-api-access-tpmbp\" (UniqueName: \"kubernetes.io/projected/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-kube-api-access-tpmbp\") pod \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") "
Nov 03 08:27:45.458336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.458311    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-plugins-config-volume\") pod \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\" (UID: \"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca\") "
Nov 03 08:27:45.458706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.458682    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-plugins-config-volume" pod "07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" (UID: "07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:45.458974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.458876    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:45.460407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.460377    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-kube-api-access-tpmbp" pod "07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" (UID: "07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca"). InnerVolumeSpecName "kube-api-access-tpmbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:45.559863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.559719    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmbp\" (UniqueName: \"kubernetes.io/projected/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca-kube-api-access-tpmbp\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:45.955522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:45.955491    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:46.122653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.122604    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" podUID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:46.123223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.123197    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" podUID="2d61781c-fa71-42ef-8ada-4350c32675bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:46.247950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.247857    6430 generic.go:356] "Generic (PLEG): container finished" podID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" containerID="267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033" exitCode=0
Nov 03 08:27:46.247950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.247912    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"
Nov 03 08:27:46.248199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.247948    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" event={"ID":"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca","Type":"ContainerDied","Data":"267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033"}
Nov 03 08:27:46.248199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.248017    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb" event={"ID":"07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca","Type":"ContainerDied","Data":"035ac6d39f605f2c485607a93f8e3bcc76500058cb8cdd04f576cc8b499e6b61"}
Nov 03 08:27:46.248199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.248043    6430 scope.go:122] "RemoveContainer" containerID="267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033"
Nov 03 08:27:46.258033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.258012    6430 scope.go:122] "RemoveContainer" containerID="267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033"
Nov 03 08:27:46.258424 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:46.258400    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033\": container with ID starting with 267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033 not found: ID does not exist" containerID="267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033"
Nov 03 08:27:46.258529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.258435    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033"} err="failed to get container status \"267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033\": rpc error: code = NotFound desc = could not find container \"267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033\": container with ID starting with 267136c50349d0dba5e00e0064b80d23080acddb58cca1103dc2265f18bd9033 not found: ID does not exist"
Nov 03 08:27:46.270735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.270712    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"]
Nov 03 08:27:46.272615 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.272596    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d6a8bf-df5c6787-gmxlb"]
Nov 03 08:27:46.617629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.617557    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" path="/var/lib/kubelet/pods/07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca/volumes"
Nov 03 08:27:46.837227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:46.837179    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:47.315944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:47.315907    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"]
Nov 03 08:27:47.436108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:47.427641    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:47.998142 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:47.994335    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:48.931411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:48.931378    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:49.730887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:49.730859    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:50.346097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:50.346044    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:50.479351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:50.479320    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:50.625642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:50.625554    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:50.829552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:50.829519    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:51.009015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.008977    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4" podUID="85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" containerName="main" containerID="cri-o://69f30cc0a7b40576c8f797602fcfea54fde22260195584bb664bc9f0098761a4" gracePeriod=45
Nov 03 08:27:51.013578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.013543    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4" podUID="85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": EOF"
Nov 03 08:27:51.282320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.282288    6430 generic.go:356] "Generic (PLEG): container finished" podID="85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" containerID="69f30cc0a7b40576c8f797602fcfea54fde22260195584bb664bc9f0098761a4" exitCode=0
Nov 03 08:27:51.282508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.282383    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4" event={"ID":"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b","Type":"ContainerDied","Data":"69f30cc0a7b40576c8f797602fcfea54fde22260195584bb664bc9f0098761a4"}
Nov 03 08:27:51.282508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.282423    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4" event={"ID":"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b","Type":"ContainerDied","Data":"3216b6bf61837c424750cde45040c502971e0a7becdf7eae41aa3dc1a9634cf3"}
Nov 03 08:27:51.282508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.282435    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3216b6bf61837c424750cde45040c502971e0a7becdf7eae41aa3dc1a9634cf3"
Nov 03 08:27:51.283561 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.283544    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:27:51.333621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.333588    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tmp-dir\") pod \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") "
Nov 03 08:27:51.333841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.333647    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-kube-api-access-bf2xx\" (UniqueName: \"kubernetes.io/projected/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-kube-api-access-bf2xx\") pod \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") "
Nov 03 08:27:51.333841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.333711    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-dshm\") pod \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") "
Nov 03 08:27:51.333841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.333797    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-model-cache\") pod \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") "
Nov 03 08:27:51.334035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.333848    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tls-certs\") pod \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") "
Nov 03 08:27:51.334035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.333878    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-home\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-home\") pod \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\" (UID: \"85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b\") "
Nov 03 08:27:51.334035 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.333903    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tmp-dir" pod "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" (UID: "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:51.334241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.334149    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-model-cache" pod "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" (UID: "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:51.334241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.334224    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-home" pod "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" (UID: "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:51.334364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.334346    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:51.334364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.334359    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:51.334482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.334368    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:51.336211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.336186    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-kube-api-access-bf2xx" pod "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" (UID: "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b"). InnerVolumeSpecName "kube-api-access-bf2xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:51.336595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.336571    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tls-certs" pod "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" (UID: "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:51.336697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.336595    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-dshm" pod "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" (UID: "85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:51.435662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.435621    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:51.435662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.435658    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf2xx\" (UniqueName: \"kubernetes.io/projected/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-kube-api-access-bf2xx\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:51.435662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.435668    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:51.574149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:51.571732    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:52.287245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:52.287212    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"
Nov 03 08:27:52.307864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:52.307840    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"]
Nov 03 08:27:52.311579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:52.311556    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0fc79b/test-model-0fc79b-kserve-79ffc49dcc-tqjk4"]
Nov 03 08:27:52.637216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:52.637006    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:52.641252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:52.641221    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" path="/var/lib/kubelet/pods/85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b/volumes"
Nov 03 08:27:53.364642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:53.364602    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:54.262394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:54.262365    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:55.241642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:55.241606    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:56.013673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:56.013607    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:56.123200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:56.123160    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" podUID="2d61781c-fa71-42ef-8ada-4350c32675bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:56.123390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:56.123172    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" podUID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:56.744912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:56.744878    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:57.451322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:57.451282    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:57.683175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:57.683138    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"]
Nov 03 08:27:57.696045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:57.696006    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"]
Nov 03 08:27:57.784011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:57.783935    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:57.899353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:57.899321    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:57.978440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:57.978318    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"]
Nov 03 08:27:57.979252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:57.979218    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv" podUID="f0c7df50-1f48-42f9-b524-52ad120683ab" containerName="maas-api" containerID="cri-o://83faf4bd87194731a4dd186919ca091a9bb3757c789ca2b39a5055f730dbfe5c" gracePeriod=30
Nov 03 08:27:58.058717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.058590    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:58.242117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.240867    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:58.330161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.330045    6430 generic.go:356] "Generic (PLEG): container finished" podID="f0c7df50-1f48-42f9-b524-52ad120683ab" containerID="83faf4bd87194731a4dd186919ca091a9bb3757c789ca2b39a5055f730dbfe5c" exitCode=0
Nov 03 08:27:58.330344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.330174    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv" event={"ID":"f0c7df50-1f48-42f9-b524-52ad120683ab","Type":"ContainerDied","Data":"83faf4bd87194731a4dd186919ca091a9bb3757c789ca2b39a5055f730dbfe5c"}
Nov 03 08:27:58.330344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.330212    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv" event={"ID":"f0c7df50-1f48-42f9-b524-52ad120683ab","Type":"ContainerDied","Data":"c9ab0ffe801b1d590f563dbd5336fc3ebae513c2eeceeda39f7a99a5887eb50e"}
Nov 03 08:27:58.330344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.330227    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ab0ffe801b1d590f563dbd5336fc3ebae513c2eeceeda39f7a99a5887eb50e"
Nov 03 08:27:58.340246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.340218    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:27:58.353346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.353317    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:58.381532 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:27:58.381495    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:27:58.462819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.462785    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-maas-api-tls\") pod \"f0c7df50-1f48-42f9-b524-52ad120683ab\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") "
Nov 03 08:27:58.463036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.462940    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c7df50-1f48-42f9-b524-52ad120683ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0c7df50-1f48-42f9-b524-52ad120683ab-openshift-service-ca\") pod \"f0c7df50-1f48-42f9-b524-52ad120683ab\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") "
Nov 03 08:27:58.463036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.462980    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-metrics-certs\") pod \"f0c7df50-1f48-42f9-b524-52ad120683ab\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") "
Nov 03 08:27:58.463036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.463015    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c7df50-1f48-42f9-b524-52ad120683ab-kube-api-access-dgvsk\" (UniqueName: \"kubernetes.io/projected/f0c7df50-1f48-42f9-b524-52ad120683ab-kube-api-access-dgvsk\") pod \"f0c7df50-1f48-42f9-b524-52ad120683ab\" (UID: \"f0c7df50-1f48-42f9-b524-52ad120683ab\") "
Nov 03 08:27:58.463416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.463386    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c7df50-1f48-42f9-b524-52ad120683ab-openshift-service-ca" pod "f0c7df50-1f48-42f9-b524-52ad120683ab" (UID: "f0c7df50-1f48-42f9-b524-52ad120683ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:58.465273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.465251    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-metrics-certs" pod "f0c7df50-1f48-42f9-b524-52ad120683ab" (UID: "f0c7df50-1f48-42f9-b524-52ad120683ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:58.465366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.465339    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-maas-api-tls" pod "f0c7df50-1f48-42f9-b524-52ad120683ab" (UID: "f0c7df50-1f48-42f9-b524-52ad120683ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:58.465422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.465407    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c7df50-1f48-42f9-b524-52ad120683ab-kube-api-access-dgvsk" pod "f0c7df50-1f48-42f9-b524-52ad120683ab" (UID: "f0c7df50-1f48-42f9-b524-52ad120683ab"). InnerVolumeSpecName "kube-api-access-dgvsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:58.469480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.469448    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:58.564976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.564865    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvsk\" (UniqueName: \"kubernetes.io/projected/f0c7df50-1f48-42f9-b524-52ad120683ab-kube-api-access-dgvsk\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:58.564976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.564908    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:58.564976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.564923    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0c7df50-1f48-42f9-b524-52ad120683ab-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:58.564976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.564935    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c7df50-1f48-42f9-b524-52ad120683ab-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:58.883711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:58.883620    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:59.329335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:59.329293    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:59.338711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:59.338669    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"
Nov 03 08:27:59.361355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:59.361259    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"]
Nov 03 08:27:59.365333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:59.365282    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-19bb0a-85db66fd6f-8pgmv"]
Nov 03 08:27:59.602136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:59.601041    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:59.737519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:59.737470    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:27:59.854845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:27:59.854767    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:00.238458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:00.238427    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:00.616299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:00.616220    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c7df50-1f48-42f9-b524-52ad120683ab" path="/var/lib/kubelet/pods/f0c7df50-1f48-42f9-b524-52ad120683ab/volumes"
Nov 03 08:28:00.753001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:00.752953    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:01.237614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:01.237564    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"]
Nov 03 08:28:01.248784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:01.248754    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"]
Nov 03 08:28:01.345987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:01.345941    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:01.464262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:01.464230    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:01.586139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:01.584521    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:02.001632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.001605    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:28:02.048757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.048733    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:28:02.124328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.124246    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d61781c-fa71-42ef-8ada-4350c32675bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d61781c-fa71-42ef-8ada-4350c32675bd-plugins-config-volume\") pod \"2d61781c-fa71-42ef-8ada-4350c32675bd\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") "
Nov 03 08:28:02.124480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.124383    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d61781c-fa71-42ef-8ada-4350c32675bd-kube-api-access-52dcl\" (UniqueName: \"kubernetes.io/projected/2d61781c-fa71-42ef-8ada-4350c32675bd-kube-api-access-52dcl\") pod \"2d61781c-fa71-42ef-8ada-4350c32675bd\" (UID: \"2d61781c-fa71-42ef-8ada-4350c32675bd\") "
Nov 03 08:28:02.124633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.124611    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d61781c-fa71-42ef-8ada-4350c32675bd-plugins-config-volume" pod "2d61781c-fa71-42ef-8ada-4350c32675bd" (UID: "2d61781c-fa71-42ef-8ada-4350c32675bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:02.124794 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.124777    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d61781c-fa71-42ef-8ada-4350c32675bd-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.126629 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.126605    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d61781c-fa71-42ef-8ada-4350c32675bd-kube-api-access-52dcl" pod "2d61781c-fa71-42ef-8ada-4350c32675bd" (UID: "2d61781c-fa71-42ef-8ada-4350c32675bd"). InnerVolumeSpecName "kube-api-access-52dcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:02.225154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.225129    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-plugins-config-volume\") pod \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") "
Nov 03 08:28:02.225298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.225245    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-kube-api-access-9b6jr\" (UniqueName: \"kubernetes.io/projected/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-kube-api-access-9b6jr\") pod \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\" (UID: \"a7b0ee03-41b9-4acf-9e37-8080afa27f1c\") "
Nov 03 08:28:02.225494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.225480    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52dcl\" (UniqueName: \"kubernetes.io/projected/2d61781c-fa71-42ef-8ada-4350c32675bd-kube-api-access-52dcl\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.225564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.225495    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-plugins-config-volume" pod "a7b0ee03-41b9-4acf-9e37-8080afa27f1c" (UID: "a7b0ee03-41b9-4acf-9e37-8080afa27f1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:02.227298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.227278    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-kube-api-access-9b6jr" pod "a7b0ee03-41b9-4acf-9e37-8080afa27f1c" (UID: "a7b0ee03-41b9-4acf-9e37-8080afa27f1c"). InnerVolumeSpecName "kube-api-access-9b6jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:02.314042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.314008    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:02.326655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.326626    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.326655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.326659    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b6jr\" (UniqueName: \"kubernetes.io/projected/a7b0ee03-41b9-4acf-9e37-8080afa27f1c-kube-api-access-9b6jr\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.350412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.349267    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr" podUID="eff2495b-3061-4d9f-a53f-78e4f9ff2adc" containerName="main" containerID="cri-o://6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329" gracePeriod=45
Nov 03 08:28:02.359977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.359949    6430 generic.go:356] "Generic (PLEG): container finished" podID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" containerID="26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e" exitCode=0
Nov 03 08:28:02.360115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.359998    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"
Nov 03 08:28:02.360115 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.360032    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" event={"ID":"a7b0ee03-41b9-4acf-9e37-8080afa27f1c","Type":"ContainerDied","Data":"26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e"}
Nov 03 08:28:02.360232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.360119    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm" event={"ID":"a7b0ee03-41b9-4acf-9e37-8080afa27f1c","Type":"ContainerDied","Data":"b8d5c63be5d77a1eb6f9cd11cf77e1c1d68f32056c81785abdaeb24105438336"}
Nov 03 08:28:02.360232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.360161    6430 scope.go:122] "RemoveContainer" containerID="26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e"
Nov 03 08:28:02.361841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.361819    6430 generic.go:356] "Generic (PLEG): container finished" podID="2d61781c-fa71-42ef-8ada-4350c32675bd" containerID="6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da" exitCode=0
Nov 03 08:28:02.361940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.361855    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"
Nov 03 08:28:02.361940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.361866    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" event={"ID":"2d61781c-fa71-42ef-8ada-4350c32675bd","Type":"ContainerDied","Data":"6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da"}
Nov 03 08:28:02.361940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.361907    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs" event={"ID":"2d61781c-fa71-42ef-8ada-4350c32675bd","Type":"ContainerDied","Data":"f2c65979138eadd947a7cf577825e2774b3e146f54ddfd86a12bcabd9dfcca27"}
Nov 03 08:28:02.389835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.389779    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr" podUID="eff2495b-3061-4d9f-a53f-78e4f9ff2adc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": dial tcp 10.133.0.168:8000: connect: connection refused"
Nov 03 08:28:02.418594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.418564    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"]
Nov 03 08:28:02.422648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.422630    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-109070-75bb7c9967-8lrzm"]
Nov 03 08:28:02.432186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.431807    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"]
Nov 03 08:28:02.440809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.440786    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-109070-77b9bd5d97-vh2qs"]
Nov 03 08:28:02.462553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.462516    6430 scope.go:122] "RemoveContainer" containerID="26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e"
Nov 03 08:28:02.463040 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:02.462998    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e\": container with ID starting with 26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e not found: ID does not exist" containerID="26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e"
Nov 03 08:28:02.463170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.463103    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e"} err="failed to get container status \"26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e\": rpc error: code = NotFound desc = could not find container \"26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e\": container with ID starting with 26dde5c99c45e6f59086ab7caa71fae2b92fab25f2ff5394766903744feaed1e not found: ID does not exist"
Nov 03 08:28:02.463170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.463149    6430 scope.go:122] "RemoveContainer" containerID="6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da"
Nov 03 08:28:02.474923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.474638    6430 scope.go:122] "RemoveContainer" containerID="6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da"
Nov 03 08:28:02.475124 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:02.475103    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da\": container with ID starting with 6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da not found: ID does not exist" containerID="6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da"
Nov 03 08:28:02.475193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.475133    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da"} err="failed to get container status \"6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da\": rpc error: code = NotFound desc = could not find container \"6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da\": container with ID starting with 6825febc605dabe1abc012a717542854b893332d2720fbeda6bc04b33b4f42da not found: ID does not exist"
Nov 03 08:28:02.603041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.603019    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:28:02.616149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.616121    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d61781c-fa71-42ef-8ada-4350c32675bd" path="/var/lib/kubelet/pods/2d61781c-fa71-42ef-8ada-4350c32675bd/volumes"
Nov 03 08:28:02.616520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.616507    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" path="/var/lib/kubelet/pods/a7b0ee03-41b9-4acf-9e37-8080afa27f1c/volumes"
Nov 03 08:28:02.684146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.684110    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" podUID="da8ccc85-c392-46c4-ab2f-9e0217071869" containerName="payload-processing" containerID="cri-o://ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7" gracePeriod=25
Nov 03 08:28:02.696691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.696625    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" podUID="df98ceaa-c373-44b9-86b6-e6141ba81776" containerName="payload-pre-processing" containerID="cri-o://561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286" gracePeriod=25
Nov 03 08:28:02.731414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731376    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tls-certs\" (UniqueName: \"kubernetes.io/secret/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tls-certs\") pod \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") "
Nov 03 08:28:02.731414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731418    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-kube-api-access-djs5d\" (UniqueName: \"kubernetes.io/projected/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-kube-api-access-djs5d\") pod \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") "
Nov 03 08:28:02.731670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731508    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tmp-dir\") pod \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") "
Nov 03 08:28:02.731670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731535    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-dshm\") pod \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") "
Nov 03 08:28:02.731670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731607    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-home\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-home\") pod \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") "
Nov 03 08:28:02.731670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731657    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-model-cache\") pod \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\" (UID: \"eff2495b-3061-4d9f-a53f-78e4f9ff2adc\") "
Nov 03 08:28:02.732152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731834    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tmp-dir" pod "eff2495b-3061-4d9f-a53f-78e4f9ff2adc" (UID: "eff2495b-3061-4d9f-a53f-78e4f9ff2adc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:02.732417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.731925    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-home" pod "eff2495b-3061-4d9f-a53f-78e4f9ff2adc" (UID: "eff2495b-3061-4d9f-a53f-78e4f9ff2adc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:02.732417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.732113    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-model-cache" pod "eff2495b-3061-4d9f-a53f-78e4f9ff2adc" (UID: "eff2495b-3061-4d9f-a53f-78e4f9ff2adc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:02.733871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.733845    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tls-certs" pod "eff2495b-3061-4d9f-a53f-78e4f9ff2adc" (UID: "eff2495b-3061-4d9f-a53f-78e4f9ff2adc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:02.733871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.733856    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-dshm" pod "eff2495b-3061-4d9f-a53f-78e4f9ff2adc" (UID: "eff2495b-3061-4d9f-a53f-78e4f9ff2adc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:02.734003 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.733862    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-kube-api-access-djs5d" pod "eff2495b-3061-4d9f-a53f-78e4f9ff2adc" (UID: "eff2495b-3061-4d9f-a53f-78e4f9ff2adc"). InnerVolumeSpecName "kube-api-access-djs5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:02.832700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.832671    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.832700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.832700    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djs5d\" (UniqueName: \"kubernetes.io/projected/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-kube-api-access-djs5d\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.832860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.832730    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.832860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.832739    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.832860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.832746    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:02.832860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:02.832754    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff2495b-3061-4d9f-a53f-78e4f9ff2adc-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.039861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.039774    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:03.370833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.370748    6430 generic.go:356] "Generic (PLEG): container finished" podID="eff2495b-3061-4d9f-a53f-78e4f9ff2adc" containerID="6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329" exitCode=0
Nov 03 08:28:03.370833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.370800    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"
Nov 03 08:28:03.370833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.370821    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr" event={"ID":"eff2495b-3061-4d9f-a53f-78e4f9ff2adc","Type":"ContainerDied","Data":"6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329"}
Nov 03 08:28:03.371476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.370860    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr" event={"ID":"eff2495b-3061-4d9f-a53f-78e4f9ff2adc","Type":"ContainerDied","Data":"547c17ada101967e5c2b048e1ca64336b996124aaadec7adb4557c78a24a7c82"}
Nov 03 08:28:03.371476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.370879    6430 scope.go:122] "RemoveContainer" containerID="6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329"
Nov 03 08:28:03.380673 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.380597    6430 scope.go:122] "RemoveContainer" containerID="6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329"
Nov 03 08:28:03.380926 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:03.380896    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329\": container with ID starting with 6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329 not found: ID does not exist" containerID="6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329"
Nov 03 08:28:03.380977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.380926    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329"} err="failed to get container status \"6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329\": rpc error: code = NotFound desc = could not find container \"6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329\": container with ID starting with 6e5207bf56b8a5a55642cfead7991385fb108e0193c2301e42e30ebd6976c329 not found: ID does not exist"
Nov 03 08:28:03.394749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.394723    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"]
Nov 03 08:28:03.402349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.402312    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0d168e/test-model-0d168e-kserve-59575f75-59frr"]
Nov 03 08:28:03.741388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:03.741357    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:04.641694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:04.641660    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff2495b-3061-4d9f-a53f-78e4f9ff2adc" path="/var/lib/kubelet/pods/eff2495b-3061-4d9f-a53f-78e4f9ff2adc/volumes"
Nov 03 08:28:04.651218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:04.651173    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:05.147592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:05.147554    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:05.325560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:05.325526    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:06.238288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.238249    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm" podUID="5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" containerName="payload-processing" containerID="cri-o://1656089d8efcf7e4fafc7fb494b26b097d903eaebb7b9ff15062e7515453ef8e" gracePeriod=25
Nov 03 08:28:06.249821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.249777    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc" podUID="0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" containerName="payload-pre-processing" containerID="cri-o://ed0f1e9726013e363a247db729534a96edf62a60fc4ddf7efb75a6bfd4ec1cbb" gracePeriod=25
Nov 03 08:28:06.265408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.265382    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:06.419085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.418502    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:06.433480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.433246    6430 generic.go:356] "Generic (PLEG): container finished" podID="0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" containerID="ed0f1e9726013e363a247db729534a96edf62a60fc4ddf7efb75a6bfd4ec1cbb" exitCode=0
Nov 03 08:28:06.433480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.433394    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc" event={"ID":"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf","Type":"ContainerDied","Data":"ed0f1e9726013e363a247db729534a96edf62a60fc4ddf7efb75a6bfd4ec1cbb"}
Nov 03 08:28:06.436140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.436110    6430 generic.go:356] "Generic (PLEG): container finished" podID="5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" containerID="1656089d8efcf7e4fafc7fb494b26b097d903eaebb7b9ff15062e7515453ef8e" exitCode=0
Nov 03 08:28:06.436382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.436364    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm" event={"ID":"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b","Type":"ContainerDied","Data":"1656089d8efcf7e4fafc7fb494b26b097d903eaebb7b9ff15062e7515453ef8e"}
Nov 03 08:28:06.519849 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:28:06.519771    6430 logging.go:55] [core] [Channel #5041 SubChannel #5042]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.106:50052", ServerName: "10.133.0.106:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.106:50052: connect: connection refused"
Nov 03 08:28:06.557935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.557905    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:28:06.566202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.566179    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:06.611318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.611289    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-kube-api-access-zwvzr\" (UniqueName: \"kubernetes.io/projected/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-kube-api-access-zwvzr\") pod \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") "
Nov 03 08:28:06.611521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.611341    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-plugins-config-volume\") pod \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\" (UID: \"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b\") "
Nov 03 08:28:06.611914 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.611885    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-plugins-config-volume" pod "5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" (UID: "5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:06.614538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.614495    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-kube-api-access-zwvzr" pod "5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" (UID: "5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b"). InnerVolumeSpecName "kube-api-access-zwvzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:06.631422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.631401    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:28:06.689238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.689200    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:06.712793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.712728    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-plugins-config-volume\") pod \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") "
Nov 03 08:28:06.713171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.713150    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-kube-api-access-wgmtp\" (UniqueName: \"kubernetes.io/projected/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-kube-api-access-wgmtp\") pod \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\" (UID: \"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf\") "
Nov 03 08:28:06.714920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.714878    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-plugins-config-volume" pod "0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" (UID: "0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:06.715340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.715311    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvzr\" (UniqueName: \"kubernetes.io/projected/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-kube-api-access-zwvzr\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:06.715443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.715346    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:06.723149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.723125    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-kube-api-access-wgmtp" pod "0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" (UID: "0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf"). InnerVolumeSpecName "kube-api-access-wgmtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:06.816931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.816841    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgmtp\" (UniqueName: \"kubernetes.io/projected/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-kube-api-access-wgmtp\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:06.816931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.816882    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:06.851408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.851233    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"]
Nov 03 08:28:06.851590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:06.851463    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" containerID="cri-o://eb03b801d5648ea65e9761ba535d41ceafad033305af9c3e8cc72f243a9ee176" gracePeriod=30
Nov 03 08:28:07.023585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.023551    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:07.279054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.279021    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:07.443268 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.443226    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm" event={"ID":"5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b","Type":"ContainerDied","Data":"fc9973fc3f06a844a5ec907f3f364e6d034515ca4fc47a1865b0cf8cb18233f8"}
Nov 03 08:28:07.443458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.443306    6430 scope.go:122] "RemoveContainer" containerID="1656089d8efcf7e4fafc7fb494b26b097d903eaebb7b9ff15062e7515453ef8e"
Nov 03 08:28:07.443458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.443242    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"
Nov 03 08:28:07.445116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.445089    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc" event={"ID":"0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf","Type":"ContainerDied","Data":"81dc8008d3174eeab59b3482fa6e8cbb69d90a7139976cc1687b1472603f28e1"}
Nov 03 08:28:07.445246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.445203    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"
Nov 03 08:28:07.455808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.455788    6430 scope.go:122] "RemoveContainer" containerID="ed0f1e9726013e363a247db729534a96edf62a60fc4ddf7efb75a6bfd4ec1cbb"
Nov 03 08:28:07.462896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.462872    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"]
Nov 03 08:28:07.466125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.466096    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm"]
Nov 03 08:28:07.479786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.479764    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"]
Nov 03 08:28:07.483695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.483671    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc"]
Nov 03 08:28:07.519712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.519652    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-178e68-86969b9cf9-9vvrc" podUID="0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.106:50052\" within 1s: context deadline exceeded"
Nov 03 08:28:07.527799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.527773    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-178e68-6745ff9d55-tp2hm" podUID="5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.107:50052\" within 1s: context deadline exceeded"
Nov 03 08:28:07.527913 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:28:07.527896    6430 logging.go:55] [core] [Channel #5043 SubChannel #5044]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.107:50052", ServerName: "10.133.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.107:50052: operation was canceled"
Nov 03 08:28:07.762528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.762494    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:07.871231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:07.871142    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:08.037538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.037498    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:08.149799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.149677    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:08.164257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.164219    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:28:08.164399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.164274    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:28:08.236358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.236320    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" podUID="da8ccc85-c392-46c4-ab2f-9e0217071869" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:08.236358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.236340    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" podUID="df98ceaa-c373-44b9-86b6-e6141ba81776" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:08.617385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.617351    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" path="/var/lib/kubelet/pods/0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf/volumes"
Nov 03 08:28:08.617757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:08.617737    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" path="/var/lib/kubelet/pods/5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b/volumes"
Nov 03 08:28:09.120141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:09.120106    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:09.912445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:09.912408    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:10.561765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:10.561735    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:11.350139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:11.350051    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:12.091935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.091900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:12.152381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.152304    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:28:12.190094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190034    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad54e90-0786-4e0b-add5-66205e0ca62b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad54e90-0786-4e0b-add5-66205e0ca62b-istiod-ca-cert\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190094 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190096    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-kube-api-access-fwt4h\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-kube-api-access-fwt4h\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190184    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-credential-socket\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190217    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-socket\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190250    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-data\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190346 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190286    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-envoy\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190348    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-token\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-token\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190389    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-podinfo\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190438    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad54e90-0786-4e0b-add5-66205e0ca62b-istiod-ca-cert" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:12.190551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190447    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-certs\") pod \"9ad54e90-0786-4e0b-add5-66205e0ca62b\" (UID: \"9ad54e90-0786-4e0b-add5-66205e0ca62b\") "
Nov 03 08:28:12.190769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190697    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-socket" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.190769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190711    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-data" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.190973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190949    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad54e90-0786-4e0b-add5-66205e0ca62b-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.191104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190988    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.191166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.191056    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.191166 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.190949    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-certs" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.191535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.191509    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-credential-socket" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.193025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.193001    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-kube-api-access-fwt4h" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "kube-api-access-fwt4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:12.193153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.193051    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-podinfo" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:28:12.193215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.193196    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-token" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:12.193691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.193670    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-envoy" pod "9ad54e90-0786-4e0b-add5-66205e0ca62b" (UID: "9ad54e90-0786-4e0b-add5-66205e0ca62b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.223054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.223024    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:12.303054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.300689    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.303054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.300733    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.303054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.300748    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.303054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.300761    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwt4h\" (UniqueName: \"kubernetes.io/projected/9ad54e90-0786-4e0b-add5-66205e0ca62b-kube-api-access-fwt4h\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.303054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.300777    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.303054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.300791    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad54e90-0786-4e0b-add5-66205e0ca62b-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.381687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.381649    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:12.420559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.420477    6430 scope.go:122] "RemoveContainer" containerID="eb03b801d5648ea65e9761ba535d41ceafad033305af9c3e8cc72f243a9ee176"
Nov 03 08:28:12.429933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.429910    6430 scope.go:122] "RemoveContainer" containerID="83faf4bd87194731a4dd186919ca091a9bb3757c789ca2b39a5055f730dbfe5c"
Nov 03 08:28:12.438815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.438796    6430 scope.go:122] "RemoveContainer" containerID="1997108fc3fd43ba6ec65162359a55763b80627f32ef17dd63bd2b60fb92e916"
Nov 03 08:28:12.510216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.506363    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:12.516741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.515002    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" event={"ID":"9ad54e90-0786-4e0b-add5-66205e0ca62b","Type":"ContainerDied","Data":"eb03b801d5648ea65e9761ba535d41ceafad033305af9c3e8cc72f243a9ee176"}
Nov 03 08:28:12.516741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.515137    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w" event={"ID":"9ad54e90-0786-4e0b-add5-66205e0ca62b","Type":"ContainerDied","Data":"06a91ae194141c649e07fc81ce305ce5d002e7f95263ddadd34b07cf64a2e0d9"}
Nov 03 08:28:12.516741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.515217    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"
Nov 03 08:28:12.540870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.540846    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"]
Nov 03 08:28:12.543538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.543512    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-c5n9w"]
Nov 03 08:28:12.616360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:12.616329    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" path="/var/lib/kubelet/pods/9ad54e90-0786-4e0b-add5-66205e0ca62b/volumes"
Nov 03 08:28:13.240230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:13.240198    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:14.216321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:14.216282    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:14.421999 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:14.421963    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:14.645340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:14.645251    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:14.792598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:14.792513    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:14.991552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:14.991476    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:15.124222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:15.124184    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:15.272672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:15.272592    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:15.953655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:15.953614    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:16.118703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:16.118668    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:16.800632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:16.800590    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:17.582183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:17.582143    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:17.687121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:17.687085    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:18.235973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.235938    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" podUID="da8ccc85-c392-46c4-ab2f-9e0217071869" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:18.236383 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.235938    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" podUID="df98ceaa-c373-44b9-86b6-e6141ba81776" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:18.339669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.339641    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:18.450537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.450501    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:18.559104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.558977    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:18.743399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.743367    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:18.851860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.851773    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:18.992489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.992452    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"]
Nov 03 08:28:18.992726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:18.992690    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr" podUID="542a29a3-ce27-4105-a01d-9ef6d4cb989f" containerName="maas-api" containerID="cri-o://06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5" gracePeriod=30
Nov 03 08:28:19.066403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.066372    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:19.234652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.234582    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:19.290654 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.290627    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:28:19.348254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.348220    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:19.381623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.381555    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542a29a3-ce27-4105-a01d-9ef6d4cb989f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542a29a3-ce27-4105-a01d-9ef6d4cb989f-openshift-service-ca\") pod \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") "
Nov 03 08:28:19.381757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.381640    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-maas-api-tls\") pod \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") "
Nov 03 08:28:19.381757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.381704    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542a29a3-ce27-4105-a01d-9ef6d4cb989f-kube-api-access-969cn\" (UniqueName: \"kubernetes.io/projected/542a29a3-ce27-4105-a01d-9ef6d4cb989f-kube-api-access-969cn\") pod \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") "
Nov 03 08:28:19.381877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.381807    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-metrics-certs\") pod \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\" (UID: \"542a29a3-ce27-4105-a01d-9ef6d4cb989f\") "
Nov 03 08:28:19.381941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.381924    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542a29a3-ce27-4105-a01d-9ef6d4cb989f-openshift-service-ca" pod "542a29a3-ce27-4105-a01d-9ef6d4cb989f" (UID: "542a29a3-ce27-4105-a01d-9ef6d4cb989f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:19.382199 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.382181    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542a29a3-ce27-4105-a01d-9ef6d4cb989f-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:19.383904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.383883    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-metrics-certs" pod "542a29a3-ce27-4105-a01d-9ef6d4cb989f" (UID: "542a29a3-ce27-4105-a01d-9ef6d4cb989f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:19.383991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.383927    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-maas-api-tls" pod "542a29a3-ce27-4105-a01d-9ef6d4cb989f" (UID: "542a29a3-ce27-4105-a01d-9ef6d4cb989f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:19.383991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.383965    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542a29a3-ce27-4105-a01d-9ef6d4cb989f-kube-api-access-969cn" pod "542a29a3-ce27-4105-a01d-9ef6d4cb989f" (UID: "542a29a3-ce27-4105-a01d-9ef6d4cb989f"). InnerVolumeSpecName "kube-api-access-969cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:19.472095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.471864    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:19.483808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.483771    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969cn\" (UniqueName: \"kubernetes.io/projected/542a29a3-ce27-4105-a01d-9ef6d4cb989f-kube-api-access-969cn\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:19.483808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.483803    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:19.483808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.483818    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542a29a3-ce27-4105-a01d-9ef6d4cb989f-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:19.567203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.567171    6430 generic.go:356] "Generic (PLEG): container finished" podID="542a29a3-ce27-4105-a01d-9ef6d4cb989f" containerID="06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5" exitCode=0
Nov 03 08:28:19.567357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.567206    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr" event={"ID":"542a29a3-ce27-4105-a01d-9ef6d4cb989f","Type":"ContainerDied","Data":"06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5"}
Nov 03 08:28:19.567357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.567246    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr" event={"ID":"542a29a3-ce27-4105-a01d-9ef6d4cb989f","Type":"ContainerDied","Data":"baea0b80649602881af2e807763ff0e49d34838fd52381b98ab38d22de9845bc"}
Nov 03 08:28:19.567357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.567271    6430 scope.go:122] "RemoveContainer" containerID="06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5"
Nov 03 08:28:19.567357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.567210    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"
Nov 03 08:28:19.577626 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.577601    6430 scope.go:122] "RemoveContainer" containerID="06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5"
Nov 03 08:28:19.577948 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:19.577925    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5\": container with ID starting with 06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5 not found: ID does not exist" containerID="06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5"
Nov 03 08:28:19.578045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.577958    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5"} err="failed to get container status \"06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5\": rpc error: code = NotFound desc = could not find container \"06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5\": container with ID starting with 06574cca99f08e93d8dcdbc25f076c081499ae3052ad883267d225c5b7cc2fb5 not found: ID does not exist"
Nov 03 08:28:19.587637 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.587614    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"]
Nov 03 08:28:19.593536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.593515    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d168e-78898d7f4c-8z9rr"]
Nov 03 08:28:19.598193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.598174    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:19.852961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:19.852930    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:20.260772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:20.260739    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:20.616014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:20.615936    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542a29a3-ce27-4105-a01d-9ef6d4cb989f" path="/var/lib/kubelet/pods/542a29a3-ce27-4105-a01d-9ef6d4cb989f/volumes"
Nov 03 08:28:21.108750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:21.108716    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:21.866320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:21.866279    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:21.968969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:21.968925    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:22.639891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:22.639853    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:22.953978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:22.953951    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:28:22.984796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:22.984760    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"]
Nov 03 08:28:22.990036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:22.990013    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:28:22.998902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:22.998873    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"]
Nov 03 08:28:23.025397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.023022    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8ccc85-c392-46c4-ab2f-9e0217071869-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ccc85-c392-46c4-ab2f-9e0217071869-plugins-config-volume\") pod \"da8ccc85-c392-46c4-ab2f-9e0217071869\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") "
Nov 03 08:28:23.025397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.023196    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8ccc85-c392-46c4-ab2f-9e0217071869-kube-api-access-b5t97\" (UniqueName: \"kubernetes.io/projected/da8ccc85-c392-46c4-ab2f-9e0217071869-kube-api-access-b5t97\") pod \"da8ccc85-c392-46c4-ab2f-9e0217071869\" (UID: \"da8ccc85-c392-46c4-ab2f-9e0217071869\") "
Nov 03 08:28:23.025397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.025241    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8ccc85-c392-46c4-ab2f-9e0217071869-plugins-config-volume" pod "da8ccc85-c392-46c4-ab2f-9e0217071869" (UID: "da8ccc85-c392-46c4-ab2f-9e0217071869"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:23.027567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.027536    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8ccc85-c392-46c4-ab2f-9e0217071869-kube-api-access-b5t97" pod "da8ccc85-c392-46c4-ab2f-9e0217071869" (UID: "da8ccc85-c392-46c4-ab2f-9e0217071869"). InnerVolumeSpecName "kube-api-access-b5t97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:23.092991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.092958    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:23.124317 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.124285    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df98ceaa-c373-44b9-86b6-e6141ba81776-kube-api-access-c9jh9\" (UniqueName: \"kubernetes.io/projected/df98ceaa-c373-44b9-86b6-e6141ba81776-kube-api-access-c9jh9\") pod \"df98ceaa-c373-44b9-86b6-e6141ba81776\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") "
Nov 03 08:28:23.124538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.124403    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df98ceaa-c373-44b9-86b6-e6141ba81776-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ceaa-c373-44b9-86b6-e6141ba81776-plugins-config-volume\") pod \"df98ceaa-c373-44b9-86b6-e6141ba81776\" (UID: \"df98ceaa-c373-44b9-86b6-e6141ba81776\") "
Nov 03 08:28:23.124641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.124627    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8ccc85-c392-46c4-ab2f-9e0217071869-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:23.124705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.124642    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5t97\" (UniqueName: \"kubernetes.io/projected/da8ccc85-c392-46c4-ab2f-9e0217071869-kube-api-access-b5t97\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:23.124811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.124790    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df98ceaa-c373-44b9-86b6-e6141ba81776-plugins-config-volume" pod "df98ceaa-c373-44b9-86b6-e6141ba81776" (UID: "df98ceaa-c373-44b9-86b6-e6141ba81776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:23.126736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.126713    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df98ceaa-c373-44b9-86b6-e6141ba81776-kube-api-access-c9jh9" pod "df98ceaa-c373-44b9-86b6-e6141ba81776" (UID: "df98ceaa-c373-44b9-86b6-e6141ba81776"). InnerVolumeSpecName "kube-api-access-c9jh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:23.202903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.202868    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:23.225675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.225643    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ceaa-c373-44b9-86b6-e6141ba81776-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:23.225675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.225670    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jh9\" (UniqueName: \"kubernetes.io/projected/df98ceaa-c373-44b9-86b6-e6141ba81776-kube-api-access-c9jh9\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:23.359498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.359467    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:23.602457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.602363    6430 generic.go:356] "Generic (PLEG): container finished" podID="da8ccc85-c392-46c4-ab2f-9e0217071869" containerID="ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7" exitCode=0
Nov 03 08:28:23.602649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.602516    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" event={"ID":"da8ccc85-c392-46c4-ab2f-9e0217071869","Type":"ContainerDied","Data":"ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7"}
Nov 03 08:28:23.602649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.602567    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x" event={"ID":"da8ccc85-c392-46c4-ab2f-9e0217071869","Type":"ContainerDied","Data":"1c443644b06313ca2c8712ce2715ea81d4ed9c183c37df340441374f1a3352e8"}
Nov 03 08:28:23.602649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.602599    6430 scope.go:122] "RemoveContainer" containerID="ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7"
Nov 03 08:28:23.602826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.602732    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"
Nov 03 08:28:23.627621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.627370    6430 generic.go:356] "Generic (PLEG): container finished" podID="df98ceaa-c373-44b9-86b6-e6141ba81776" containerID="561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286" exitCode=0
Nov 03 08:28:23.627621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.627439    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" event={"ID":"df98ceaa-c373-44b9-86b6-e6141ba81776","Type":"ContainerDied","Data":"561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286"}
Nov 03 08:28:23.627621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.627535    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp" event={"ID":"df98ceaa-c373-44b9-86b6-e6141ba81776","Type":"ContainerDied","Data":"5a1dbe60ecc12858daf7f3642c4a065cfd4ef8485afeaa9db75af8d41fade35b"}
Nov 03 08:28:23.628177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.628052    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"
Nov 03 08:28:23.651547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.651521    6430 scope.go:122] "RemoveContainer" containerID="ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7"
Nov 03 08:28:23.651922 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:23.651894    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7\": container with ID starting with ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7 not found: ID does not exist" containerID="ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7"
Nov 03 08:28:23.651985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.651937    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7"} err="failed to get container status \"ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7\": rpc error: code = NotFound desc = could not find container \"ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7\": container with ID starting with ab028b5a5f39d8ec01047e00928e1ece28f79a611accb083e03228060dbb72f7 not found: ID does not exist"
Nov 03 08:28:23.651985 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.651963    6430 scope.go:122] "RemoveContainer" containerID="561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286"
Nov 03 08:28:23.661796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.661775    6430 scope.go:122] "RemoveContainer" containerID="561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286"
Nov 03 08:28:23.662137 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:23.662110    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286\": container with ID starting with 561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286 not found: ID does not exist" containerID="561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286"
Nov 03 08:28:23.662247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.662142    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286"} err="failed to get container status \"561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286\": rpc error: code = NotFound desc = could not find container \"561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286\": container with ID starting with 561e314844751beb8c142054f45dbd38e44c4a870f37adcdfa556c0dadc43286 not found: ID does not exist"
Nov 03 08:28:23.676031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.676003    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"]
Nov 03 08:28:23.698974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.698946    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0fc79b-55d5f449d7-jr72x"]
Nov 03 08:28:23.712316 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.712288    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"]
Nov 03 08:28:23.722390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.722363    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0fc79b-59b6c5d78c-qwpzp"]
Nov 03 08:28:23.814735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.814696    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"]
Nov 03 08:28:23.815119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:23.815087    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy" containerID="cri-o://c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b" gracePeriod=30
Nov 03 08:28:24.186054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:24.186011    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:24.615909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:24.615829    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8ccc85-c392-46c4-ab2f-9e0217071869" path="/var/lib/kubelet/pods/da8ccc85-c392-46c4-ab2f-9e0217071869/volumes"
Nov 03 08:28:24.616257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:24.616243    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df98ceaa-c373-44b9-86b6-e6141ba81776" path="/var/lib/kubelet/pods/df98ceaa-c373-44b9-86b6-e6141ba81776/volumes"
Nov 03 08:28:24.902143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:24.896046    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:25.391110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.388565    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"]
Nov 03 08:28:25.391110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.388818    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7" podUID="b9e014ac-f28f-4861-a732-c5321d301811" containerName="maas-api" containerID="cri-o://221eb41a428412ad57e952a269061141c2f798f501cb5b590ddc4eee88dbe9fe" gracePeriod=30
Nov 03 08:28:25.456491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.456435    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:25.641725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.641606    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:25.653370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.653335    6430 generic.go:356] "Generic (PLEG): container finished" podID="b9e014ac-f28f-4861-a732-c5321d301811" containerID="221eb41a428412ad57e952a269061141c2f798f501cb5b590ddc4eee88dbe9fe" exitCode=0
Nov 03 08:28:25.653526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.653416    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7" event={"ID":"b9e014ac-f28f-4861-a732-c5321d301811","Type":"ContainerDied","Data":"221eb41a428412ad57e952a269061141c2f798f501cb5b590ddc4eee88dbe9fe"}
Nov 03 08:28:25.685261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.685235    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:28:25.795807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.795776    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:25.851463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.851434    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e014ac-f28f-4861-a732-c5321d301811-kube-api-access-pp6nl\" (UniqueName: \"kubernetes.io/projected/b9e014ac-f28f-4861-a732-c5321d301811-kube-api-access-pp6nl\") pod \"b9e014ac-f28f-4861-a732-c5321d301811\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") "
Nov 03 08:28:25.851627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.851510    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e014ac-f28f-4861-a732-c5321d301811-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e014ac-f28f-4861-a732-c5321d301811-openshift-service-ca\") pod \"b9e014ac-f28f-4861-a732-c5321d301811\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") "
Nov 03 08:28:25.851627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.851556    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-maas-api-tls\") pod \"b9e014ac-f28f-4861-a732-c5321d301811\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") "
Nov 03 08:28:25.851756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.851646    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-metrics-certs\") pod \"b9e014ac-f28f-4861-a732-c5321d301811\" (UID: \"b9e014ac-f28f-4861-a732-c5321d301811\") "
Nov 03 08:28:25.851919 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.851894    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e014ac-f28f-4861-a732-c5321d301811-openshift-service-ca" pod "b9e014ac-f28f-4861-a732-c5321d301811" (UID: "b9e014ac-f28f-4861-a732-c5321d301811"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:25.853935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.853913    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-metrics-certs" pod "b9e014ac-f28f-4861-a732-c5321d301811" (UID: "b9e014ac-f28f-4861-a732-c5321d301811"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:25.854050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.853963    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e014ac-f28f-4861-a732-c5321d301811-kube-api-access-pp6nl" pod "b9e014ac-f28f-4861-a732-c5321d301811" (UID: "b9e014ac-f28f-4861-a732-c5321d301811"). InnerVolumeSpecName "kube-api-access-pp6nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:25.854050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.853966    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-maas-api-tls" pod "b9e014ac-f28f-4861-a732-c5321d301811" (UID: "b9e014ac-f28f-4861-a732-c5321d301811"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:25.902556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.902456    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:25.953471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.953441    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e014ac-f28f-4861-a732-c5321d301811-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:25.953471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.953465    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:25.953471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.953474    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e014ac-f28f-4861-a732-c5321d301811-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:25.953689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:25.953484    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp6nl\" (UniqueName: \"kubernetes.io/projected/b9e014ac-f28f-4861-a732-c5321d301811-kube-api-access-pp6nl\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:26.243737 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.243704    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:26.659859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.659831    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"
Nov 03 08:28:26.660272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.659874    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7" event={"ID":"b9e014ac-f28f-4861-a732-c5321d301811","Type":"ContainerDied","Data":"3e2bed458bd60f02cf7927c382b97f1a803351bbe3e3c0c46e2bfe63cdc33483"}
Nov 03 08:28:26.660272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.659935    6430 scope.go:122] "RemoveContainer" containerID="221eb41a428412ad57e952a269061141c2f798f501cb5b590ddc4eee88dbe9fe"
Nov 03 08:28:26.673649 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.673627    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"]
Nov 03 08:28:26.684399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.684374    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ed5d1a-6f8df67bcc-7cqd7"]
Nov 03 08:28:26.736414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.736387    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:26.849008 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:26.848973    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:27.056189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:27.056152    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:27.512441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:27.512407    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:27.621476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:27.621444    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:27.985698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:27.985641    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz" podUID="fad3bdc8-1b67-411b-8842-903becf9baf4" containerName="payload-processing" containerID="cri-o://ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840" gracePeriod=25
Nov 03 08:28:27.999343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:27.999311    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj" podUID="184484fb-ece1-4e09-91fd-d6261a76b6eb" containerName="payload-pre-processing" containerID="cri-o://435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b" gracePeriod=25
Nov 03 08:28:28.263916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.263888    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:28:28.290298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.290274    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:28:28.377634 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:28.375209    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:28:28.382233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382206    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184484fb-ece1-4e09-91fd-d6261a76b6eb-kube-api-access-rjv8z\" (UniqueName: \"kubernetes.io/projected/184484fb-ece1-4e09-91fd-d6261a76b6eb-kube-api-access-rjv8z\") pod \"184484fb-ece1-4e09-91fd-d6261a76b6eb\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") "
Nov 03 08:28:28.382357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382253    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad3bdc8-1b67-411b-8842-903becf9baf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad3bdc8-1b67-411b-8842-903becf9baf4-plugins-config-volume\") pod \"fad3bdc8-1b67-411b-8842-903becf9baf4\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") "
Nov 03 08:28:28.382450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382432    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad3bdc8-1b67-411b-8842-903becf9baf4-kube-api-access-m5ln2\" (UniqueName: \"kubernetes.io/projected/fad3bdc8-1b67-411b-8842-903becf9baf4-kube-api-access-m5ln2\") pod \"fad3bdc8-1b67-411b-8842-903becf9baf4\" (UID: \"fad3bdc8-1b67-411b-8842-903becf9baf4\") "
Nov 03 08:28:28.382512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382496    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184484fb-ece1-4e09-91fd-d6261a76b6eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184484fb-ece1-4e09-91fd-d6261a76b6eb-plugins-config-volume\") pod \"184484fb-ece1-4e09-91fd-d6261a76b6eb\" (UID: \"184484fb-ece1-4e09-91fd-d6261a76b6eb\") "
Nov 03 08:28:28.382661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382639    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad3bdc8-1b67-411b-8842-903becf9baf4-plugins-config-volume" pod "fad3bdc8-1b67-411b-8842-903becf9baf4" (UID: "fad3bdc8-1b67-411b-8842-903becf9baf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:28.382902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382830    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184484fb-ece1-4e09-91fd-d6261a76b6eb-plugins-config-volume" pod "184484fb-ece1-4e09-91fd-d6261a76b6eb" (UID: "184484fb-ece1-4e09-91fd-d6261a76b6eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:28.382902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382960    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184484fb-ece1-4e09-91fd-d6261a76b6eb-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:28.382902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.382974    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad3bdc8-1b67-411b-8842-903becf9baf4-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:28.384517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.384495    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184484fb-ece1-4e09-91fd-d6261a76b6eb-kube-api-access-rjv8z" pod "184484fb-ece1-4e09-91fd-d6261a76b6eb" (UID: "184484fb-ece1-4e09-91fd-d6261a76b6eb"). InnerVolumeSpecName "kube-api-access-rjv8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:28.384624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.384592    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad3bdc8-1b67-411b-8842-903becf9baf4-kube-api-access-m5ln2" pod "fad3bdc8-1b67-411b-8842-903becf9baf4" (UID: "fad3bdc8-1b67-411b-8842-903becf9baf4"). InnerVolumeSpecName "kube-api-access-m5ln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:28.483837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.483798    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ln2\" (UniqueName: \"kubernetes.io/projected/fad3bdc8-1b67-411b-8842-903becf9baf4-kube-api-access-m5ln2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:28.483837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.483831    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjv8z\" (UniqueName: \"kubernetes.io/projected/184484fb-ece1-4e09-91fd-d6261a76b6eb-kube-api-access-rjv8z\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:28.623656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.623555    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e014ac-f28f-4861-a732-c5321d301811" path="/var/lib/kubelet/pods/b9e014ac-f28f-4861-a732-c5321d301811/volumes"
Nov 03 08:28:28.671499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.671461    6430 generic.go:356] "Generic (PLEG): container finished" podID="fad3bdc8-1b67-411b-8842-903becf9baf4" containerID="ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840" exitCode=0
Nov 03 08:28:28.671718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.671529    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"
Nov 03 08:28:28.671718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.671549    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz" event={"ID":"fad3bdc8-1b67-411b-8842-903becf9baf4","Type":"ContainerDied","Data":"ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840"}
Nov 03 08:28:28.671718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.671625    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz" event={"ID":"fad3bdc8-1b67-411b-8842-903becf9baf4","Type":"ContainerDied","Data":"1fcd895171f2e22caadecd7e6bfae43d4ffa4e24d833aa7d28d1099f7a47b6a3"}
Nov 03 08:28:28.671718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.671657    6430 scope.go:122] "RemoveContainer" containerID="ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840"
Nov 03 08:28:28.672945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.672924    6430 generic.go:356] "Generic (PLEG): container finished" podID="184484fb-ece1-4e09-91fd-d6261a76b6eb" containerID="435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b" exitCode=0
Nov 03 08:28:28.673092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.673000    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"
Nov 03 08:28:28.673092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.673014    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj" event={"ID":"184484fb-ece1-4e09-91fd-d6261a76b6eb","Type":"ContainerDied","Data":"435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b"}
Nov 03 08:28:28.673188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.673091    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj" event={"ID":"184484fb-ece1-4e09-91fd-d6261a76b6eb","Type":"ContainerDied","Data":"bf66a1d9451a074284b06725b51a844f79612e9ed0c881e946144d8ebf2b5f2b"}
Nov 03 08:28:28.681215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.681195    6430 scope.go:122] "RemoveContainer" containerID="ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840"
Nov 03 08:28:28.681498 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:28.681465    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840\": container with ID starting with ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840 not found: ID does not exist" containerID="ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840"
Nov 03 08:28:28.681498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.681491    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840"} err="failed to get container status \"ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840\": rpc error: code = NotFound desc = could not find container \"ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840\": container with ID starting with ec524f3317c5fa1e49d9930be43060d30c102733ab320bebad339fd79e7a6840 not found: ID does not exist"
Nov 03 08:28:28.681605 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.681506    6430 scope.go:122] "RemoveContainer" containerID="435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b"
Nov 03 08:28:28.688880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.688855    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"]
Nov 03 08:28:28.690401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.690377    6430 scope.go:122] "RemoveContainer" containerID="435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b"
Nov 03 08:28:28.690690 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:28.690670    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b\": container with ID starting with 435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b not found: ID does not exist" containerID="435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b"
Nov 03 08:28:28.690759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.690701    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b"} err="failed to get container status \"435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b\": rpc error: code = NotFound desc = could not find container \"435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b\": container with ID starting with 435ed018f54aaf23fc4c14f1663d7610d2fdb98d875f3661d8135ca73d7e938b not found: ID does not exist"
Nov 03 08:28:28.694777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.694751    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed5d1a-74d4744d8c-jczvz"]
Nov 03 08:28:28.703896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.703860    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"]
Nov 03 08:28:28.707486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.707460    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed5d1a-75684c65bc-45ntj"]
Nov 03 08:28:28.718525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:28.718495    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:29.089735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.089700    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:28:29.190960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.190921    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-certs\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.190997    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-token\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-token\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191028    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-credential-socket\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191112    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-kube-api-access-lxgts\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-kube-api-access-lxgts\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191142    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b27641-47ac-4a12-9a06-96b6b02240b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b27641-47ac-4a12-9a06-96b6b02240b8-istiod-ca-cert\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191177    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-socket\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191328    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-envoy\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191366    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-certs" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:29.191451 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191423    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-podinfo\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191448    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-credential-socket" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:29.191680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191460    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-socket" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:29.191680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191463    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-data\") pod \"08b27641-47ac-4a12-9a06-96b6b02240b8\" (UID: \"08b27641-47ac-4a12-9a06-96b6b02240b8\") "
Nov 03 08:28:29.191680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191582    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b27641-47ac-4a12-9a06-96b6b02240b8-istiod-ca-cert" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:29.191861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191702    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-data" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:29.191950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191925    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.192039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191951    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.192039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191966    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.192039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191979    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b27641-47ac-4a12-9a06-96b6b02240b8-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.192039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.191989    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.193514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.193486    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-token" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:29.193801 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.193784    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-podinfo" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:28:29.194051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.194036    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-kube-api-access-lxgts" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "kube-api-access-lxgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:29.194153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.194046    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-envoy" pod "08b27641-47ac-4a12-9a06-96b6b02240b8" (UID: "08b27641-47ac-4a12-9a06-96b6b02240b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:29.293536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.293504    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.293536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.293534    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxgts\" (UniqueName: \"kubernetes.io/projected/08b27641-47ac-4a12-9a06-96b6b02240b8-kube-api-access-lxgts\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.293536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.293545    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.293536 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.293553    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b27641-47ac-4a12-9a06-96b6b02240b8-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:29.558697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.558611    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:29.679223 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.679186    6430 generic.go:356] "Generic (PLEG): container finished" podID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerID="c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b" exitCode=0
Nov 03 08:28:29.679441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.679248    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"
Nov 03 08:28:29.679441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.679268    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" event={"ID":"08b27641-47ac-4a12-9a06-96b6b02240b8","Type":"ContainerDied","Data":"c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b"}
Nov 03 08:28:29.679441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.679325    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7" event={"ID":"08b27641-47ac-4a12-9a06-96b6b02240b8","Type":"ContainerDied","Data":"76e1da6336a77bac3685e53c37b3548858a52848433691a44c04fe056ecc3646"}
Nov 03 08:28:29.679441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.679354    6430 scope.go:122] "RemoveContainer" containerID="c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b"
Nov 03 08:28:29.688942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.688923    6430 scope.go:122] "RemoveContainer" containerID="c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b"
Nov 03 08:28:29.689265 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:29.689245    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b\": container with ID starting with c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b not found: ID does not exist" containerID="c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b"
Nov 03 08:28:29.689331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.689274    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b"} err="failed to get container status \"c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b\": rpc error: code = NotFound desc = could not find container \"c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b\": container with ID starting with c6598b802f2a8e8a86ae852edc753d74fb58d48d511c0981c7b1c791c5b6278b not found: ID does not exist"
Nov 03 08:28:29.700638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.700614    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"]
Nov 03 08:28:29.702940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:29.702916    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bm2g7"]
Nov 03 08:28:30.260174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:30.260142    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:30.615869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:30.615794    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" path="/var/lib/kubelet/pods/08b27641-47ac-4a12-9a06-96b6b02240b8/volumes"
Nov 03 08:28:30.616201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:30.616187    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184484fb-ece1-4e09-91fd-d6261a76b6eb" path="/var/lib/kubelet/pods/184484fb-ece1-4e09-91fd-d6261a76b6eb/volumes"
Nov 03 08:28:30.616534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:30.616522    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad3bdc8-1b67-411b-8842-903becf9baf4" path="/var/lib/kubelet/pods/fad3bdc8-1b67-411b-8842-903becf9baf4/volumes"
Nov 03 08:28:30.965728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:30.965696    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:31.417356 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:31.417274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:31.758994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:31.758952    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:32.449392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:32.449358    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"]
Nov 03 08:28:32.455641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:32.455616    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"]
Nov 03 08:28:32.528475 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:32.528446    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:32.654796 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:32.654766    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:32.800220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:32.798749    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"]
Nov 03 08:28:32.800220 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:32.799887    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy" containerID="cri-o://23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da" gracePeriod=30
Nov 03 08:28:32.872539 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:32.872505    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:33.320947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:33.320909    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:33.451509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:33.451469    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:34.449773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:34.449735    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:35.257152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:35.257114    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:35.952804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:35.952772    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:36.649931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:36.649898    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:37.460251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:37.460216    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:37.864159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:37.864127    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:37.970487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:37.970447    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:38.081358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.081295    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:28:38.179126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179050    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-socket\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179148    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-token\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-token\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179197    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-certs\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179232    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-podinfo\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179267    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-credential-socket\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179336    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-envoy\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179358    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-socket" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:38.179511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179387    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-kube-api-access-4vgv8\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-kube-api-access-4vgv8\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179437    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-data\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179511 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179474    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7352fb60-2c9e-4777-8d2a-22aa012189af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352fb60-2c9e-4777-8d2a-22aa012189af-istiod-ca-cert\") pod \"7352fb60-2c9e-4777-8d2a-22aa012189af\" (UID: \"7352fb60-2c9e-4777-8d2a-22aa012189af\") "
Nov 03 08:28:38.179791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179562    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-certs" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:38.179791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179746    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-credential-socket" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:38.179927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179876    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.179927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179899    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.179927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179912    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.179927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179881    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7352fb60-2c9e-4777-8d2a-22aa012189af-istiod-ca-cert" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:38.179927 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.179923    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-data" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:38.181696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.181674    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-envoy" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:38.181833 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.181703    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-kube-api-access-4vgv8" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "kube-api-access-4vgv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:38.182106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.182088    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-podinfo" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:28:38.182106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.182097    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-token" pod "7352fb60-2c9e-4777-8d2a-22aa012189af" (UID: "7352fb60-2c9e-4777-8d2a-22aa012189af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:38.195256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.195229    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:38.281042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.281003    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.281042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.281036    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.281042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.281046    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.281327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.281055    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vgv8\" (UniqueName: \"kubernetes.io/projected/7352fb60-2c9e-4777-8d2a-22aa012189af-kube-api-access-4vgv8\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.281327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.281098    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352fb60-2c9e-4777-8d2a-22aa012189af-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.281327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.281111    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352fb60-2c9e-4777-8d2a-22aa012189af-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:38.315608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.315567    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"]
Nov 03 08:28:38.327940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.327905    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"]
Nov 03 08:28:38.423264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.423171    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:38.731363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.731320    6430 generic.go:356] "Generic (PLEG): container finished" podID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerID="23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da" exitCode=0
Nov 03 08:28:38.731548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.731417    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"
Nov 03 08:28:38.731608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.731549    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" event={"ID":"7352fb60-2c9e-4777-8d2a-22aa012189af","Type":"ContainerDied","Data":"23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da"}
Nov 03 08:28:38.731662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.731625    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf" event={"ID":"7352fb60-2c9e-4777-8d2a-22aa012189af","Type":"ContainerDied","Data":"4a2309589261e30dbf874753042b4e8e0baa66b6fe61cae63ce53e7fcdf4d070"}
Nov 03 08:28:38.731662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.731648    6430 scope.go:122] "RemoveContainer" containerID="23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da"
Nov 03 08:28:38.742111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.742089    6430 scope.go:122] "RemoveContainer" containerID="300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a"
Nov 03 08:28:38.747011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.746988    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"]
Nov 03 08:28:38.752905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.752885    6430 scope.go:122] "RemoveContainer" containerID="23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da"
Nov 03 08:28:38.753073 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.753037    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed5d1a-openshift-default-9bb5b4489-n6hpf"]
Nov 03 08:28:38.753224 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:38.753204    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da\": container with ID starting with 23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da not found: ID does not exist" containerID="23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da"
Nov 03 08:28:38.753266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.753234    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da"} err="failed to get container status \"23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da\": rpc error: code = NotFound desc = could not find container \"23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da\": container with ID starting with 23f35b2b12a22d3c734486eade862c084ebd69f99e81f57aa3efa01cbd70f8da not found: ID does not exist"
Nov 03 08:28:38.753266 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.753251    6430 scope.go:122] "RemoveContainer" containerID="300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a"
Nov 03 08:28:38.753523 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:38.753502    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a\": container with ID starting with 300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a not found: ID does not exist" containerID="300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a"
Nov 03 08:28:38.753582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:38.753532    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a"} err="failed to get container status \"300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a\": rpc error: code = NotFound desc = could not find container \"300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a\": container with ID starting with 300ace5338fc0733eedda9f76402e926e124c213443e36f82c49a021ab28105a not found: ID does not exist"
Nov 03 08:28:39.049039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.048960    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:39.573995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.573953    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"]
Nov 03 08:28:39.574528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574511    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574531    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574547    6430 cpu_manager.go:414] "containerMap: removing container" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574553    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574559    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d0fd224e-e7a1-46be-89cf-3fae1f1120e2" containerName="payload-processing"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574564    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fd224e-e7a1-46be-89cf-3fae1f1120e2" containerName="payload-processing"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574570    6430 cpu_manager.go:414] "containerMap: removing container" podUID="94702209-a919-4445-9577-cd5739032003" containerName="maas-api"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574574    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94702209-a919-4445-9577-cd5739032003" containerName="maas-api"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574581    6430 cpu_manager.go:414] "containerMap: removing container" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574585    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574593    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy"
Nov 03 08:28:39.574596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574598    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574604    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574608    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574615    6430 cpu_manager.go:414] "containerMap: removing container" podUID="df98ceaa-c373-44b9-86b6-e6141ba81776" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574620    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df98ceaa-c373-44b9-86b6-e6141ba81776" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574627    6430 cpu_manager.go:414] "containerMap: removing container" podUID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574633    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574639    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574644    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574650    6430 cpu_manager.go:414] "containerMap: removing container" podUID="04905656-0d4f-427d-a89f-9eba8e9a17b0" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574654    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04905656-0d4f-427d-a89f-9eba8e9a17b0" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574661    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574665    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574672    6430 cpu_manager.go:414] "containerMap: removing container" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574675    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574684    6430 cpu_manager.go:414] "containerMap: removing container" podUID="85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574688    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574694    6430 cpu_manager.go:414] "containerMap: removing container" podUID="88feb840-2575-4c86-a9c2-b526ad23c84c" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574700    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88feb840-2575-4c86-a9c2-b526ad23c84c" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574706    6430 cpu_manager.go:414] "containerMap: removing container" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574710    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d9bbf7-e4ea-4e03-b1b6-10de49ad8cda" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574719    6430 cpu_manager.go:414] "containerMap: removing container" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574723    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574730    6430 cpu_manager.go:414] "containerMap: removing container" podUID="89fd0430-b072-458b-91fa-8a1459789c95" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574734    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fd0430-b072-458b-91fa-8a1459789c95" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574741    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574745    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574750    6430 cpu_manager.go:414] "containerMap: removing container" podUID="607fca89-8aa0-4714-820a-393ab37084c0" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574754    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607fca89-8aa0-4714-820a-393ab37084c0" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574761    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f79c787b-4338-4a84-9433-ead2be648e61" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574765    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79c787b-4338-4a84-9433-ead2be648e61" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574771    6430 cpu_manager.go:414] "containerMap: removing container" podUID="51912a37-547f-4db7-b5e9-a56b62cfa3bf" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574776    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51912a37-547f-4db7-b5e9-a56b62cfa3bf" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574782    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574786    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574791    6430 cpu_manager.go:414] "containerMap: removing container" podUID="42e9de3e-9a77-4f01-980e-72497ced47b4" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574795    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e9de3e-9a77-4f01-980e-72497ced47b4" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574802    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574806    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574813    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574817    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574824    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574828    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574834    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a9cf911f-3512-4745-a08b-bf6616df4894" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574837    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cf911f-3512-4745-a08b-bf6616df4894" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574843    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1b7c09ad-494e-412e-9d98-093a3e5df344" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574848    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7c09ad-494e-412e-9d98-093a3e5df344" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574857    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574861    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574867    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574872    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574879    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f0c7df50-1f48-42f9-b524-52ad120683ab" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574883    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c7df50-1f48-42f9-b524-52ad120683ab" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574889    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9368968e-6637-402a-998f-8075b5dddfb7" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574894    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9368968e-6637-402a-998f-8075b5dddfb7" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574900    6430 cpu_manager.go:414] "containerMap: removing container" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574904    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574911    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2387acc4-0876-413d-9714-aafb39f01ee6" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574916    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2387acc4-0876-413d-9714-aafb39f01ee6" containerName="main"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574922    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574928    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" containerName="payload-pre-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574934    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4bde8834-c6f6-44d8-8df3-542c155b7315" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574938    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bde8834-c6f6-44d8-8df3-542c155b7315" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574945    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574950    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574955    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fad3bdc8-1b67-411b-8842-903becf9baf4" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574960    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad3bdc8-1b67-411b-8842-903becf9baf4" containerName="payload-processing"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574966    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b9e014ac-f28f-4861-a732-c5321d301811" containerName="maas-api"
Nov 03 08:28:39.574940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574970    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e014ac-f28f-4861-a732-c5321d301811" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574976    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574980    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574986    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574990    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.574996    6430 cpu_manager.go:414] "containerMap: removing container" podUID="184484fb-ece1-4e09-91fd-d6261a76b6eb" containerName="payload-pre-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575000    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184484fb-ece1-4e09-91fd-d6261a76b6eb" containerName="payload-pre-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575008    6430 cpu_manager.go:414] "containerMap: removing container" podUID="eff2495b-3061-4d9f-a53f-78e4f9ff2adc" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575013    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff2495b-3061-4d9f-a53f-78e4f9ff2adc" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575018    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575022    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575029    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2d61781c-fa71-42ef-8ada-4350c32675bd" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575033    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d61781c-fa71-42ef-8ada-4350c32675bd" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575038    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8857534d-237b-4ff9-bf5b-b18cf35727f8" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575042    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8857534d-237b-4ff9-bf5b-b18cf35727f8" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575048    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575052    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575071    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575076    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575082    6430 cpu_manager.go:414] "containerMap: removing container" podUID="31bab6fc-0528-47ae-836e-743075206e82" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575086    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bab6fc-0528-47ae-836e-743075206e82" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575092    6430 cpu_manager.go:414] "containerMap: removing container" podUID="542a29a3-ce27-4105-a01d-9ef6d4cb989f" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575096    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542a29a3-ce27-4105-a01d-9ef6d4cb989f" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575104    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575108    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575114    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575118    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575123    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575128    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575134    6430 cpu_manager.go:414] "containerMap: removing container" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575138    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575144    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575148    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575152    6430 cpu_manager.go:414] "containerMap: removing container" podUID="da8ccc85-c392-46c4-ab2f-9e0217071869" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575156    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8ccc85-c392-46c4-ab2f-9e0217071869" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575165    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575169    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575227    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d001284-d2f0-4e54-af47-26b578dc8596" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575236    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cf911f-3512-4745-a08b-bf6616df4894" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575241    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7c09ad-494e-412e-9d98-093a3e5df344" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575248    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad54e90-0786-4e0b-add5-66205e0ca62b" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575253    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35a857c-1664-4657-ab11-9b5ee33baa24" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575258    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c7df50-1f48-42f9-b524-52ad120683ab" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575263    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9368968e-6637-402a-998f-8075b5dddfb7" containerName="payload-pre-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575269    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="83537c3e-e723-4150-a413-c817d9630222" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575274    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2387acc4-0876-413d-9714-aafb39f01ee6" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575279    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b0ee03-41b9-4acf-9e37-8080afa27f1c" containerName="payload-pre-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575284    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bde8834-c6f6-44d8-8df3-542c155b7315" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575289    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22dcaae-af57-4cd0-b304-fa20192b9a62" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575293    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad3bdc8-1b67-411b-8842-903becf9baf4" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575298    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e014ac-f28f-4861-a732-c5321d301811" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575302    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57e36a8-937e-44f0-b243-a95a341901a0" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575306    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa1bb45-7443-4bb4-a5f1-0b01ac2b0c6b" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575310    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="184484fb-ece1-4e09-91fd-d6261a76b6eb" containerName="payload-pre-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575316    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff2495b-3061-4d9f-a53f-78e4f9ff2adc" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575320    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d648aabf-a0e9-476b-a8d4-35efc9a9aaa6" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575325    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d61781c-fa71-42ef-8ada-4350c32675bd" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575329    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8857534d-237b-4ff9-bf5b-b18cf35727f8" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575338    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abdfcc3-5815-49a3-8459-6324e5013873" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575343    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce14510d-b9f4-4d27-9f20-38ca6d0cbf8f" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575348    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bab6fc-0528-47ae-836e-743075206e82" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575352    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="542a29a3-ce27-4105-a01d-9ef6d4cb989f" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575358    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba467c2-f014-4052-8ebc-e769cb311877" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575362    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca64b314-7012-47d5-9e3e-361468b2b02e" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575367    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575372    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="35388dcd-6433-4f50-a084-947740876acb" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575376    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee9f9ca-9623-4300-9b04-6c3ba857841d" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575381    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8ccc85-c392-46c4-ab2f-9e0217071869" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575387    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575391    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb640fd-a708-4f4f-8607-ec9e414ad41a" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575396    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="709e5b39-cbd8-4b8d-84e7-fb2643d2f4cd" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575402    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fd224e-e7a1-46be-89cf-3fae1f1120e2" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575406    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="94702209-a919-4445-9577-cd5739032003" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575411    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="feacf2a1-9660-4dc5-beec-3406a32e24fe" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575416    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43b9ca2-2706-4449-8102-55b7b146379e" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575422    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575427    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="df98ceaa-c373-44b9-86b6-e6141ba81776" containerName="payload-pre-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575432    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fc0cdf-c4e1-4cd5-addb-ab4a4c6646ca" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575437    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3df2b48-9c28-4c34-bde0-b7e35e55f41f" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575442    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="04905656-0d4f-427d-a89f-9eba8e9a17b0" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575446    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f954b829-91a3-4aee-925c-363f1cfa9824" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575450    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba4e9c9-971e-4be9-80db-813f7e1a70ee" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575455    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e330a8-56b8-40b8-83aa-9bb1dfbb4b3b" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575460    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="88feb840-2575-4c86-a9c2-b526ad23c84c" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575465    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b27641-47ac-4a12-9a06-96b6b02240b8" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575470    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fd0430-b072-458b-91fa-8a1459789c95" containerName="payload-processing"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575475    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="607fca89-8aa0-4714-820a-393ab37084c0" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575481    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79c787b-4338-4a84-9433-ead2be648e61" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575487    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="51912a37-547f-4db7-b5e9-a56b62cfa3bf" containerName="maas-api"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575492    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d752af-c375-437c-bde9-7886542bf533" containerName="istio-proxy"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575498    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e9de3e-9a77-4f01-980e-72497ced47b4" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575502    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0ad9a5-cbf2-4ae1-8527-ea74dfdd9938" containerName="main"
Nov 03 08:28:39.577054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.575507    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb7cd2c-544b-4947-a5ce-3fdbbb0d2bcf" containerName="payload-pre-processing"
Nov 03 08:28:39.580042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.580022    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.582899 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.582870    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:28:39.587868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.587843    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"]
Nov 03 08:28:39.683827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.683804    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:28:39.695683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.695654    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.695797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.695730    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.695915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.695896    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszc2\" (UniqueName: \"kubernetes.io/projected/b220302f-02f5-43e4-96df-e41e25536e8c-kube-api-access-jszc2\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.797092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.797052    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.797247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.797126    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszc2\" (UniqueName: \"kubernetes.io/projected/b220302f-02f5-43e4-96df-e41e25536e8c-kube-api-access-jszc2\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.797247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.797167    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.797785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.797730    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.799360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.799334    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.803566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.803543    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszc2\" (UniqueName: \"kubernetes.io/projected/b220302f-02f5-43e4-96df-e41e25536e8c-kube-api-access-jszc2\") pod \"maas-api-revoke-sub-849668f1-8zxgr\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:39.836515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.836462    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:39.896169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:39.896137    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:40.169730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:40.169657    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"]
Nov 03 08:28:40.173278 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:28:40.173252    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb220302f_02f5_43e4_96df_e41e25536e8c.slice/crio-33aa6a564c58a5959cba1c1a07276e656a9ae869000d028fb09769fb31c3ffc6 WatchSource:0}: Error finding container 33aa6a564c58a5959cba1c1a07276e656a9ae869000d028fb09769fb31c3ffc6: Status 404 returned error can't find the container with id 33aa6a564c58a5959cba1c1a07276e656a9ae869000d028fb09769fb31c3ffc6
Nov 03 08:28:40.563258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:40.563223    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:40.615643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:40.615611    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7352fb60-2c9e-4777-8d2a-22aa012189af" path="/var/lib/kubelet/pods/7352fb60-2c9e-4777-8d2a-22aa012189af/volumes"
Nov 03 08:28:40.744283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:40.744250    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr" event={"ID":"b220302f-02f5-43e4-96df-e41e25536e8c","Type":"ContainerStarted","Data":"f751bbfa2c981fe2c584b9433637d0299850111dfd8792318d70beccc7863d68"}
Nov 03 08:28:40.744440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:40.744302    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr" event={"ID":"b220302f-02f5-43e4-96df-e41e25536e8c","Type":"ContainerStarted","Data":"33aa6a564c58a5959cba1c1a07276e656a9ae869000d028fb09769fb31c3ffc6"}
Nov 03 08:28:40.761256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:40.761168    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr" podStartSLOduration=1.7611541819999998 podStartE2EDuration="1.761154182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:28:40.75983102 +0000 UTC m=+2070.262666533" watchObservedRunningTime="2026-11-03 08:28:40.761154182 +0000 UTC m=+2070.263989720"
Nov 03 08:28:41.298754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:41.298635    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:41.750344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:41.750312    6430 generic.go:356] "Generic (PLEG): container finished" podID="b220302f-02f5-43e4-96df-e41e25536e8c" containerID="f751bbfa2c981fe2c584b9433637d0299850111dfd8792318d70beccc7863d68" exitCode=0
Nov 03 08:28:41.750498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:41.750391    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr" event={"ID":"b220302f-02f5-43e4-96df-e41e25536e8c","Type":"ContainerDied","Data":"f751bbfa2c981fe2c584b9433637d0299850111dfd8792318d70beccc7863d68"}
Nov 03 08:28:42.078353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:42.078259    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:42.896602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:42.896579    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:43.029998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.029905    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-xks-ca\") pod \"b220302f-02f5-43e4-96df-e41e25536e8c\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") "
Nov 03 08:28:43.029998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.029974    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-openshift-ca\") pod \"b220302f-02f5-43e4-96df-e41e25536e8c\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") "
Nov 03 08:28:43.030518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.030092    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b220302f-02f5-43e4-96df-e41e25536e8c-kube-api-access-jszc2\" (UniqueName: \"kubernetes.io/projected/b220302f-02f5-43e4-96df-e41e25536e8c-kube-api-access-jszc2\") pod \"b220302f-02f5-43e4-96df-e41e25536e8c\" (UID: \"b220302f-02f5-43e4-96df-e41e25536e8c\") "
Nov 03 08:28:43.030685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.030658    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-openshift-ca" pod "b220302f-02f5-43e4-96df-e41e25536e8c" (UID: "b220302f-02f5-43e4-96df-e41e25536e8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:43.032842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.032813    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b220302f-02f5-43e4-96df-e41e25536e8c-kube-api-access-jszc2" pod "b220302f-02f5-43e4-96df-e41e25536e8c" (UID: "b220302f-02f5-43e4-96df-e41e25536e8c"). InnerVolumeSpecName "kube-api-access-jszc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:43.033327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.032826    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-xks-ca" pod "b220302f-02f5-43e4-96df-e41e25536e8c" (UID: "b220302f-02f5-43e4-96df-e41e25536e8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:43.071957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.071925    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:43.131345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.131306    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:43.131345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.131341    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220302f-02f5-43e4-96df-e41e25536e8c-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:43.131345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.131351    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jszc2\" (UniqueName: \"kubernetes.io/projected/b220302f-02f5-43e4-96df-e41e25536e8c-kube-api-access-jszc2\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:43.701256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.701222    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:43.761990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.761953    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"
Nov 03 08:28:43.761990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.761962    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr" event={"ID":"b220302f-02f5-43e4-96df-e41e25536e8c","Type":"ContainerDied","Data":"33aa6a564c58a5959cba1c1a07276e656a9ae869000d028fb09769fb31c3ffc6"}
Nov 03 08:28:43.762299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.762016    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33aa6a564c58a5959cba1c1a07276e656a9ae869000d028fb09769fb31c3ffc6"
Nov 03 08:28:43.811778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.811745    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"]
Nov 03 08:28:43.812195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.812178    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b220302f-02f5-43e4-96df-e41e25536e8c" containerName="revoke-keys"
Nov 03 08:28:43.812260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.812197    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b220302f-02f5-43e4-96df-e41e25536e8c" containerName="revoke-keys"
Nov 03 08:28:43.812295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.812259    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b220302f-02f5-43e4-96df-e41e25536e8c" containerName="revoke-keys"
Nov 03 08:28:43.812295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.812275    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8e8ea0-340a-4aef-8b26-ee5b79d47394" containerName="istio-proxy"
Nov 03 08:28:43.815534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.815508    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:43.823615 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.823583    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:28:43.823733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.823633    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:28:43.836821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.836782    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"]
Nov 03 08:28:43.842184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.842041    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:43.939763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.939717    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg5lh\" (UniqueName: \"kubernetes.io/projected/5aff661b-468f-4669-a378-bde2349dcab3-kube-api-access-hg5lh\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:43.940283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.939917    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:43.940283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:43.939977    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.041486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.040834    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg5lh\" (UniqueName: \"kubernetes.io/projected/5aff661b-468f-4669-a378-bde2349dcab3-kube-api-access-hg5lh\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.041486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.040957    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.041486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.040991    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.041981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.041744    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.043593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.043569    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.048127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.048101    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg5lh\" (UniqueName: \"kubernetes.io/projected/5aff661b-468f-4669-a378-bde2349dcab3-kube-api-access-hg5lh\") pod \"maas-api-revoke-sub-90a57fc5-pcgwr\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.129528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.129501    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:44.402976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.402881    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"]
Nov 03 08:28:44.407542 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:28:44.407512    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aff661b_468f_4669_a378_bde2349dcab3.slice/crio-fbe6e7070bc7433da647e4b08a7442a42a26d59a98333cb0aed0ca467d1b48de WatchSource:0}: Error finding container fbe6e7070bc7433da647e4b08a7442a42a26d59a98333cb0aed0ca467d1b48de: Status 404 returned error can't find the container with id fbe6e7070bc7433da647e4b08a7442a42a26d59a98333cb0aed0ca467d1b48de
Nov 03 08:28:44.767893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.767858    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr" event={"ID":"5aff661b-468f-4669-a378-bde2349dcab3","Type":"ContainerStarted","Data":"575967de530b420c16da7dec75bb11a9764102066ff49ebbe404c6b71ae65e6f"}
Nov 03 08:28:44.768087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.767908    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr" event={"ID":"5aff661b-468f-4669-a378-bde2349dcab3","Type":"ContainerStarted","Data":"fbe6e7070bc7433da647e4b08a7442a42a26d59a98333cb0aed0ca467d1b48de"}
Nov 03 08:28:44.784122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:44.784074    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr" podStartSLOduration=1.784044776 podStartE2EDuration="1.784044776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:28:44.782323277 +0000 UTC m=+2074.285158791" watchObservedRunningTime="2026-11-03 08:28:44.784044776 +0000 UTC m=+2074.286880282"
Nov 03 08:28:45.774709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:45.774673    6430 generic.go:356] "Generic (PLEG): container finished" podID="5aff661b-468f-4669-a378-bde2349dcab3" containerID="575967de530b420c16da7dec75bb11a9764102066ff49ebbe404c6b71ae65e6f" exitCode=0
Nov 03 08:28:45.775119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:45.774766    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr" event={"ID":"5aff661b-468f-4669-a378-bde2349dcab3","Type":"ContainerDied","Data":"575967de530b420c16da7dec75bb11a9764102066ff49ebbe404c6b71ae65e6f"}
Nov 03 08:28:46.913791 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:46.913760    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:47.069379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.069311    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aff661b-468f-4669-a378-bde2349dcab3-kube-api-access-hg5lh\" (UniqueName: \"kubernetes.io/projected/5aff661b-468f-4669-a378-bde2349dcab3-kube-api-access-hg5lh\") pod \"5aff661b-468f-4669-a378-bde2349dcab3\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") "
Nov 03 08:28:47.069379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.069346    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-xks-ca\") pod \"5aff661b-468f-4669-a378-bde2349dcab3\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") "
Nov 03 08:28:47.069614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.069464    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-openshift-ca\") pod \"5aff661b-468f-4669-a378-bde2349dcab3\" (UID: \"5aff661b-468f-4669-a378-bde2349dcab3\") "
Nov 03 08:28:47.069822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.069801    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-openshift-ca" pod "5aff661b-468f-4669-a378-bde2349dcab3" (UID: "5aff661b-468f-4669-a378-bde2349dcab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:47.071480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.071463    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aff661b-468f-4669-a378-bde2349dcab3-kube-api-access-hg5lh" pod "5aff661b-468f-4669-a378-bde2349dcab3" (UID: "5aff661b-468f-4669-a378-bde2349dcab3"). InnerVolumeSpecName "kube-api-access-hg5lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:47.071621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.071608    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-xks-ca" pod "5aff661b-468f-4669-a378-bde2349dcab3" (UID: "5aff661b-468f-4669-a378-bde2349dcab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:47.170086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.170040    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.170086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.170084    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg5lh\" (UniqueName: \"kubernetes.io/projected/5aff661b-468f-4669-a378-bde2349dcab3-kube-api-access-hg5lh\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.170227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.170095    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aff661b-468f-4669-a378-bde2349dcab3-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.470227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.470181    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main" containerID="cri-o://b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4" gracePeriod=45
Nov 03 08:28:47.481491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.481461    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" containerID="cri-o://2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad" gracePeriod=45
Nov 03 08:28:47.496195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.496167    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Nov 03 08:28:47.712596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.712572    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:28:47.740038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.740016    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:28:47.787030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.786995    6430 generic.go:356] "Generic (PLEG): container finished" podID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerID="b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4" exitCode=0
Nov 03 08:28:47.787238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.787117    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" event={"ID":"dcc94a01-1373-403b-b51c-cc7789cab2d2","Type":"ContainerDied","Data":"b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4"}
Nov 03 08:28:47.787238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.787146    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"
Nov 03 08:28:47.787238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.787179    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78" event={"ID":"dcc94a01-1373-403b-b51c-cc7789cab2d2","Type":"ContainerDied","Data":"84a577b700f40ad2108adf81d213b12c7e74303c147885c2a46f3bfc1faffe93"}
Nov 03 08:28:47.787238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.787203    6430 scope.go:122] "RemoveContainer" containerID="b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4"
Nov 03 08:28:47.789237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.789168    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr" event={"ID":"5aff661b-468f-4669-a378-bde2349dcab3","Type":"ContainerDied","Data":"fbe6e7070bc7433da647e4b08a7442a42a26d59a98333cb0aed0ca467d1b48de"}
Nov 03 08:28:47.789237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.789204    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe6e7070bc7433da647e4b08a7442a42a26d59a98333cb0aed0ca467d1b48de"
Nov 03 08:28:47.789416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.789257    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"
Nov 03 08:28:47.793735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.793690    6430 generic.go:356] "Generic (PLEG): container finished" podID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerID="2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad" exitCode=0
Nov 03 08:28:47.793897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.793751    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" event={"ID":"d4194f3d-d12c-417b-a715-7fedf4dfd732","Type":"ContainerDied","Data":"2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad"}
Nov 03 08:28:47.793897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.793781    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29" event={"ID":"d4194f3d-d12c-417b-a715-7fedf4dfd732","Type":"ContainerDied","Data":"a9d613d679f12784c1b359ca7ee6409b89331fd15a366e318fc5a00a080e5acd"}
Nov 03 08:28:47.794110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.794055    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"
Nov 03 08:28:47.797228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.797204    6430 scope.go:122] "RemoveContainer" containerID="b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4"
Nov 03 08:28:47.797485 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:47.797460    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4\": container with ID starting with b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4 not found: ID does not exist" containerID="b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4"
Nov 03 08:28:47.797559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.797496    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4"} err="failed to get container status \"b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4\": rpc error: code = NotFound desc = could not find container \"b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4\": container with ID starting with b1c8066f367ceefb58e8111385777eb28503633c34ba0a59560950fdcf75f1e4 not found: ID does not exist"
Nov 03 08:28:47.797559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.797521    6430 scope.go:122] "RemoveContainer" containerID="2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad"
Nov 03 08:28:47.808592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.808574    6430 scope.go:122] "RemoveContainer" containerID="2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad"
Nov 03 08:28:47.808867 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:47.808849    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad\": container with ID starting with 2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad not found: ID does not exist" containerID="2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad"
Nov 03 08:28:47.808924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.808878    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad"} err="failed to get container status \"2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad\": rpc error: code = NotFound desc = could not find container \"2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad\": container with ID starting with 2d910a3a5c302fbebe3571f58f1b306c42f3f490522d960bf090221c93e26aad not found: ID does not exist"
Nov 03 08:28:47.877045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877005    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-model-cache\") pod \"d4194f3d-d12c-417b-a715-7fedf4dfd732\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") "
Nov 03 08:28:47.877237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877103    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-tmp-dir\") pod \"dcc94a01-1373-403b-b51c-cc7789cab2d2\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") "
Nov 03 08:28:47.877290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877246    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-model-cache\") pod \"dcc94a01-1373-403b-b51c-cc7789cab2d2\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") "
Nov 03 08:28:47.877337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877306    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-home\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-home\") pod \"dcc94a01-1373-403b-b51c-cc7789cab2d2\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") "
Nov 03 08:28:47.877379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877333    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-model-cache" pod "d4194f3d-d12c-417b-a715-7fedf4dfd732" (UID: "d4194f3d-d12c-417b-a715-7fedf4dfd732"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.877379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877346    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-tmp-dir" pod "dcc94a01-1373-403b-b51c-cc7789cab2d2" (UID: "dcc94a01-1373-403b-b51c-cc7789cab2d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.877379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877346    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-dshm\") pod \"d4194f3d-d12c-417b-a715-7fedf4dfd732\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") "
Nov 03 08:28:47.877515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877409    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc94a01-1373-403b-b51c-cc7789cab2d2-kube-api-access-fv5rv\" (UniqueName: \"kubernetes.io/projected/dcc94a01-1373-403b-b51c-cc7789cab2d2-kube-api-access-fv5rv\") pod \"dcc94a01-1373-403b-b51c-cc7789cab2d2\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") "
Nov 03 08:28:47.877515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877430    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4194f3d-d12c-417b-a715-7fedf4dfd732-kube-api-access-8fnts\" (UniqueName: \"kubernetes.io/projected/d4194f3d-d12c-417b-a715-7fedf4dfd732-kube-api-access-8fnts\") pod \"d4194f3d-d12c-417b-a715-7fedf4dfd732\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") "
Nov 03 08:28:47.877515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877465    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc94a01-1373-403b-b51c-cc7789cab2d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc94a01-1373-403b-b51c-cc7789cab2d2-tls-certs\") pod \"dcc94a01-1373-403b-b51c-cc7789cab2d2\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") "
Nov 03 08:28:47.877515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877494    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-dshm\") pod \"dcc94a01-1373-403b-b51c-cc7789cab2d2\" (UID: \"dcc94a01-1373-403b-b51c-cc7789cab2d2\") "
Nov 03 08:28:47.877727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877539    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-tmp-dir\") pod \"d4194f3d-d12c-417b-a715-7fedf4dfd732\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") "
Nov 03 08:28:47.877727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877535    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-model-cache" pod "dcc94a01-1373-403b-b51c-cc7789cab2d2" (UID: "dcc94a01-1373-403b-b51c-cc7789cab2d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.877727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877543    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-home" pod "dcc94a01-1373-403b-b51c-cc7789cab2d2" (UID: "dcc94a01-1373-403b-b51c-cc7789cab2d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.877884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877758    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-tmp-dir" pod "d4194f3d-d12c-417b-a715-7fedf4dfd732" (UID: "d4194f3d-d12c-417b-a715-7fedf4dfd732"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.877884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877806    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-home\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-home\") pod \"d4194f3d-d12c-417b-a715-7fedf4dfd732\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") "
Nov 03 08:28:47.877884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.877853    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4194f3d-d12c-417b-a715-7fedf4dfd732-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4194f3d-d12c-417b-a715-7fedf4dfd732-tls-certs\") pod \"d4194f3d-d12c-417b-a715-7fedf4dfd732\" (UID: \"d4194f3d-d12c-417b-a715-7fedf4dfd732\") "
Nov 03 08:28:47.878248 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.878076    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-home" pod "d4194f3d-d12c-417b-a715-7fedf4dfd732" (UID: "d4194f3d-d12c-417b-a715-7fedf4dfd732"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.878248 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.878254    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.878490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.878277    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.878490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.878293    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.878490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.878306    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.878490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.878319    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.879988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.879958    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc94a01-1373-403b-b51c-cc7789cab2d2-tls-certs" pod "dcc94a01-1373-403b-b51c-cc7789cab2d2" (UID: "dcc94a01-1373-403b-b51c-cc7789cab2d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:47.880206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.880185    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-dshm" pod "d4194f3d-d12c-417b-a715-7fedf4dfd732" (UID: "d4194f3d-d12c-417b-a715-7fedf4dfd732"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.880414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.880384    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc94a01-1373-403b-b51c-cc7789cab2d2-kube-api-access-fv5rv" pod "dcc94a01-1373-403b-b51c-cc7789cab2d2" (UID: "dcc94a01-1373-403b-b51c-cc7789cab2d2"). InnerVolumeSpecName "kube-api-access-fv5rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:47.880608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.880586    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4194f3d-d12c-417b-a715-7fedf4dfd732-tls-certs" pod "d4194f3d-d12c-417b-a715-7fedf4dfd732" (UID: "d4194f3d-d12c-417b-a715-7fedf4dfd732"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:47.880677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.880639    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4194f3d-d12c-417b-a715-7fedf4dfd732-kube-api-access-8fnts" pod "d4194f3d-d12c-417b-a715-7fedf4dfd732" (UID: "d4194f3d-d12c-417b-a715-7fedf4dfd732"). InnerVolumeSpecName "kube-api-access-8fnts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:47.880766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.880753    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-dshm" pod "dcc94a01-1373-403b-b51c-cc7789cab2d2" (UID: "dcc94a01-1373-403b-b51c-cc7789cab2d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.979375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.979335    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.979375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.979367    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv5rv\" (UniqueName: \"kubernetes.io/projected/dcc94a01-1373-403b-b51c-cc7789cab2d2-kube-api-access-fv5rv\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.979375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.979380    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fnts\" (UniqueName: \"kubernetes.io/projected/d4194f3d-d12c-417b-a715-7fedf4dfd732-kube-api-access-8fnts\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.979871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.979394    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc94a01-1373-403b-b51c-cc7789cab2d2-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.979871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.979406    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc94a01-1373-403b-b51c-cc7789cab2d2-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.979871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.979416    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4194f3d-d12c-417b-a715-7fedf4dfd732-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.979871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:47.979428    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4194f3d-d12c-417b-a715-7fedf4dfd732-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:48.106417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:48.106392    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"]
Nov 03 08:28:48.108687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:48.108664    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-2-w5-ffe7a8-kserve-748ddc4f8-ltd78"]
Nov 03 08:28:48.117499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:48.117476    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"]
Nov 03 08:28:48.123148 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:48.123126    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/distinct-w5-ffe7a8-kserve-5c7d69696f-fbb29"]
Nov 03 08:28:48.615740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:48.615703    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" path="/var/lib/kubelet/pods/d4194f3d-d12c-417b-a715-7fedf4dfd732/volumes"
Nov 03 08:28:48.616230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:48.616210    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" path="/var/lib/kubelet/pods/dcc94a01-1373-403b-b51c-cc7789cab2d2/volumes"
Nov 03 08:28:53.337623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.337528    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main" containerID="cri-o://bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017" gracePeriod=45
Nov 03 08:28:53.349388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.349334    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" containerID="cri-o://dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f" gracePeriod=45
Nov 03 08:28:53.496718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.496679    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": dial tcp 10.133.0.36:8000: connect: connection refused"
Nov 03 08:28:53.608719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.608693    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:28:53.613356 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.612691    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631022    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tls-certs\" (UniqueName: \"kubernetes.io/secret/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tls-certs\") pod \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631110    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-model-cache\") pod \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631157    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-home\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-home\") pod \"50c5eff3-19ce-4d3e-a475-0ac713d19978\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631210    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-home\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-home\") pod \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631245    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-model-cache\") pod \"50c5eff3-19ce-4d3e-a475-0ac713d19978\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631302    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c5eff3-19ce-4d3e-a475-0ac713d19978-kube-api-access-gzzlg\" (UniqueName: \"kubernetes.io/projected/50c5eff3-19ce-4d3e-a475-0ac713d19978-kube-api-access-gzzlg\") pod \"50c5eff3-19ce-4d3e-a475-0ac713d19978\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631340    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tmp-dir\") pod \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631376    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-kube-api-access-9flgx\" (UniqueName: \"kubernetes.io/projected/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-kube-api-access-9flgx\") pod \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") "
Nov 03 08:28:53.631397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631402    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-dshm\") pod \"50c5eff3-19ce-4d3e-a475-0ac713d19978\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") "
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631421    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-model-cache" pod "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" (UID: "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631438    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c5eff3-19ce-4d3e-a475-0ac713d19978-tls-certs\" (UniqueName: \"kubernetes.io/secret/50c5eff3-19ce-4d3e-a475-0ac713d19978-tls-certs\") pod \"50c5eff3-19ce-4d3e-a475-0ac713d19978\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") "
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631512    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-tmp-dir\") pod \"50c5eff3-19ce-4d3e-a475-0ac713d19978\" (UID: \"50c5eff3-19ce-4d3e-a475-0ac713d19978\") "
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631571    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-dshm\") pod \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\" (UID: \"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838\") "
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631639    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tmp-dir" pod "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" (UID: "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631881    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-home" pod "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" (UID: "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631957    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.631986    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.632092    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-model-cache" pod "50c5eff3-19ce-4d3e-a475-0ac713d19978" (UID: "50c5eff3-19ce-4d3e-a475-0ac713d19978"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.632233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.632110    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-tmp-dir" pod "50c5eff3-19ce-4d3e-a475-0ac713d19978" (UID: "50c5eff3-19ce-4d3e-a475-0ac713d19978"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.632954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.632459    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-home" pod "50c5eff3-19ce-4d3e-a475-0ac713d19978" (UID: "50c5eff3-19ce-4d3e-a475-0ac713d19978"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.635721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.635676    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-dshm" pod "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" (UID: "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.636215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.636094    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-dshm" pod "50c5eff3-19ce-4d3e-a475-0ac713d19978" (UID: "50c5eff3-19ce-4d3e-a475-0ac713d19978"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.636215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.636173    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-kube-api-access-9flgx" pod "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" (UID: "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838"). InnerVolumeSpecName "kube-api-access-9flgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:53.636911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.636888    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c5eff3-19ce-4d3e-a475-0ac713d19978-kube-api-access-gzzlg" pod "50c5eff3-19ce-4d3e-a475-0ac713d19978" (UID: "50c5eff3-19ce-4d3e-a475-0ac713d19978"). InnerVolumeSpecName "kube-api-access-gzzlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:53.637481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.637454    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c5eff3-19ce-4d3e-a475-0ac713d19978-tls-certs" pod "50c5eff3-19ce-4d3e-a475-0ac713d19978" (UID: "50c5eff3-19ce-4d3e-a475-0ac713d19978"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:53.638929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.638897    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tls-certs" pod "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" (UID: "b08503f1-9bb1-4a0f-b1ab-09f6e8e21838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:53.732616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732579    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732615    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732628    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732640    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732653    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732666    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732678    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzzlg\" (UniqueName: \"kubernetes.io/projected/50c5eff3-19ce-4d3e-a475-0ac713d19978-kube-api-access-gzzlg\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732692    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flgx\" (UniqueName: \"kubernetes.io/projected/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838-kube-api-access-9flgx\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732704    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c5eff3-19ce-4d3e-a475-0ac713d19978-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.732850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.732718    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c5eff3-19ce-4d3e-a475-0ac713d19978-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.823337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.823302    6430 generic.go:356] "Generic (PLEG): container finished" podID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerID="bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017" exitCode=0
Nov 03 08:28:53.823512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.823344    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"
Nov 03 08:28:53.823512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.823378    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" event={"ID":"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838","Type":"ContainerDied","Data":"bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017"}
Nov 03 08:28:53.823512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.823419    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6" event={"ID":"b08503f1-9bb1-4a0f-b1ab-09f6e8e21838","Type":"ContainerDied","Data":"d4bb5ada08488cc3cb4988620fc5d2365cc591d7b1ead1ace18146776a8ff4cb"}
Nov 03 08:28:53.823512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.823435    6430 scope.go:122] "RemoveContainer" containerID="bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017"
Nov 03 08:28:53.824974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.824950    6430 generic.go:356] "Generic (PLEG): container finished" podID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerID="dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f" exitCode=0
Nov 03 08:28:53.825136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.824986    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" event={"ID":"50c5eff3-19ce-4d3e-a475-0ac713d19978","Type":"ContainerDied","Data":"dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f"}
Nov 03 08:28:53.825136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.824992    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"
Nov 03 08:28:53.825136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.825014    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr" event={"ID":"50c5eff3-19ce-4d3e-a475-0ac713d19978","Type":"ContainerDied","Data":"079ed550e0af72491b336fd44492f031f85b6b4422cb352c552d060989080906"}
Nov 03 08:28:53.832576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.832555    6430 scope.go:122] "RemoveContainer" containerID="bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017"
Nov 03 08:28:53.832855 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:53.832831    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017\": container with ID starting with bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017 not found: ID does not exist" containerID="bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017"
Nov 03 08:28:53.832948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.832868    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017"} err="failed to get container status \"bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017\": rpc error: code = NotFound desc = could not find container \"bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017\": container with ID starting with bf555b0f183ca2f442d33a0646c5820fdde0b4a459019744e90e35c72a175017 not found: ID does not exist"
Nov 03 08:28:53.832948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.832892    6430 scope.go:122] "RemoveContainer" containerID="dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f"
Nov 03 08:28:53.841143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.841123    6430 scope.go:122] "RemoveContainer" containerID="dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f"
Nov 03 08:28:53.841411 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:53.841392    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f\": container with ID starting with dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f not found: ID does not exist" containerID="dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f"
Nov 03 08:28:53.841520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.841420    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f"} err="failed to get container status \"dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f\": rpc error: code = NotFound desc = could not find container \"dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f\": container with ID starting with dbe56017c8236108fac352e103a21b0c7d7d99059bad8e8a916cba476e325e5f not found: ID does not exist"
Nov 03 08:28:53.845211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.845161    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"]
Nov 03 08:28:53.849990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.849967    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/premium-w2-6bc553-kserve-7f684958bd-9pnx6"]
Nov 03 08:28:53.866306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.866281    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"]
Nov 03 08:28:53.869905 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:53.869879    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/simulator-w2-6bc553-kserve-56b57887b5-gwnqr"]
Nov 03 08:28:54.619265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:54.619164    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" path="/var/lib/kubelet/pods/50c5eff3-19ce-4d3e-a475-0ac713d19978/volumes"
Nov 03 08:28:54.620041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:54.619782    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" path="/var/lib/kubelet/pods/b08503f1-9bb1-4a0f-b1ab-09f6e8e21838/volumes"
Nov 03 08:28:54.691870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:54.691840    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:54.864792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:54.864752    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:54.975579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:54.975546    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:55.473193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:55.473151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:55.908448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:55.908352    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"]
Nov 03 08:28:55.908979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:55.908585    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx" podUID="fc5c9dff-5472-4849-add5-af54751535e9" containerName="maas-api" containerID="cri-o://a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7" gracePeriod=30
Nov 03 08:28:55.987566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:55.987534    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:56.152469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.152424    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:56.161569 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.161522    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:28:56.258928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.258901    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5c9dff-5472-4849-add5-af54751535e9-kube-api-access-bjmxx\" (UniqueName: \"kubernetes.io/projected/fc5c9dff-5472-4849-add5-af54751535e9-kube-api-access-bjmxx\") pod \"fc5c9dff-5472-4849-add5-af54751535e9\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") "
Nov 03 08:28:56.259134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.258952    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5c9dff-5472-4849-add5-af54751535e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c9dff-5472-4849-add5-af54751535e9-openshift-service-ca\") pod \"fc5c9dff-5472-4849-add5-af54751535e9\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") "
Nov 03 08:28:56.259134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.259011    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-maas-api-tls\") pod \"fc5c9dff-5472-4849-add5-af54751535e9\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") "
Nov 03 08:28:56.259134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.259110    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-metrics-certs\") pod \"fc5c9dff-5472-4849-add5-af54751535e9\" (UID: \"fc5c9dff-5472-4849-add5-af54751535e9\") "
Nov 03 08:28:56.259432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.259409    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5c9dff-5472-4849-add5-af54751535e9-openshift-service-ca" pod "fc5c9dff-5472-4849-add5-af54751535e9" (UID: "fc5c9dff-5472-4849-add5-af54751535e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:56.261399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.261372    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-metrics-certs" pod "fc5c9dff-5472-4849-add5-af54751535e9" (UID: "fc5c9dff-5472-4849-add5-af54751535e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:56.261505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.261441    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5c9dff-5472-4849-add5-af54751535e9-kube-api-access-bjmxx" pod "fc5c9dff-5472-4849-add5-af54751535e9" (UID: "fc5c9dff-5472-4849-add5-af54751535e9"). InnerVolumeSpecName "kube-api-access-bjmxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:56.261505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.261451    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-maas-api-tls" pod "fc5c9dff-5472-4849-add5-af54751535e9" (UID: "fc5c9dff-5472-4849-add5-af54751535e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:56.262751 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.262718    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:56.359987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.359947    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:56.359987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.359981    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmxx\" (UniqueName: \"kubernetes.io/projected/fc5c9dff-5472-4849-add5-af54751535e9-kube-api-access-bjmxx\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:56.359987 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.359992    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c9dff-5472-4849-add5-af54751535e9-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:56.360274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.360000    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c9dff-5472-4849-add5-af54751535e9-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:56.751983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.751955    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:28:56.862176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.862148    6430 generic.go:356] "Generic (PLEG): container finished" podID="fc5c9dff-5472-4849-add5-af54751535e9" containerID="a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7" exitCode=0
Nov 03 08:28:56.862322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.862192    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"
Nov 03 08:28:56.862322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.862232    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx" event={"ID":"fc5c9dff-5472-4849-add5-af54751535e9","Type":"ContainerDied","Data":"a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7"}
Nov 03 08:28:56.862322 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.862292    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx" event={"ID":"fc5c9dff-5472-4849-add5-af54751535e9","Type":"ContainerDied","Data":"1bd8c4d47166b85af7ee96bf4be02f0d91b8537d9d3e8158c03b2dc85d7fb0cf"}
Nov 03 08:28:56.862431 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.862322    6430 scope.go:122] "RemoveContainer" containerID="a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7"
Nov 03 08:28:56.871407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.871389    6430 scope.go:122] "RemoveContainer" containerID="a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7"
Nov 03 08:28:56.871678 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:56.871646    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7\": container with ID starting with a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7 not found: ID does not exist" containerID="a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7"
Nov 03 08:28:56.871777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.871685    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7"} err="failed to get container status \"a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7\": rpc error: code = NotFound desc = could not find container \"a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7\": container with ID starting with a101667636b49caab2f051a1bda76d37eb05ce2a17cca47b9362df9f6b89cfc7 not found: ID does not exist"
Nov 03 08:28:56.875721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.875673    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"]
Nov 03 08:28:56.878046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:56.878025    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6bc553-7cf6b8d567-s45hx"]
Nov 03 08:28:58.374196 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:28:58.374150    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:28:58.615147 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:28:58.615115    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5c9dff-5472-4849-add5-af54751535e9" path="/var/lib/kubelet/pods/fc5c9dff-5472-4849-add5-af54751535e9/volumes"
Nov 03 08:29:01.361593 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:01.361557    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:01.480671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:01.480636    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:01.640218 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:01.640124    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"]
Nov 03 08:29:01.640709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:01.640675    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy" containerID="cri-o://c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d" gracePeriod=30
Nov 03 08:29:01.699378 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:01.699334    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:02.931641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:02.931603    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:03.066823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:03.066789    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:03.180545 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:03.180501    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"]
Nov 03 08:29:03.180761 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:03.180737    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy" containerID="cri-o://0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca" gracePeriod=30
Nov 03 08:29:03.245523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:03.245425    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:06.886587 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.886555    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:29:06.914202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.914126    6430 generic.go:356] "Generic (PLEG): container finished" podID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerID="c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d" exitCode=0
Nov 03 08:29:06.914202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.914178    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"
Nov 03 08:29:06.914202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.914187    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" event={"ID":"f0510f54-3454-46df-b6ee-cbb7b53dfb8c","Type":"ContainerDied","Data":"c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d"}
Nov 03 08:29:06.914453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.914240    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh" event={"ID":"f0510f54-3454-46df-b6ee-cbb7b53dfb8c","Type":"ContainerDied","Data":"660a357c6573be2c511cb2151628b8632433bc6e8eb44ee298d74497a99757e8"}
Nov 03 08:29:06.914453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.914268    6430 scope.go:122] "RemoveContainer" containerID="c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d"
Nov 03 08:29:06.924739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.924719    6430 scope.go:122] "RemoveContainer" containerID="c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d"
Nov 03 08:29:06.925171 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:29:06.925146    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d\": container with ID starting with c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d not found: ID does not exist" containerID="c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d"
Nov 03 08:29:06.925295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.925181    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d"} err="failed to get container status \"c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d\": rpc error: code = NotFound desc = could not find container \"c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d\": container with ID starting with c3e4988b56466848ede29595696c63752532755768ba386818f02a4c6f072c4d not found: ID does not exist"
Nov 03 08:29:06.949160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949136    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-kube-api-access-rv2md\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-kube-api-access-rv2md\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949188    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-socket\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949231    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-token\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-token\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949297    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-data\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949339    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istiod-ca-cert\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949365    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-podinfo\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949408    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-credential-socket\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949522    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-socket" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:06.949557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949529    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-certs\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949582    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-envoy\") pod \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\" (UID: \"f0510f54-3454-46df-b6ee-cbb7b53dfb8c\") "
Nov 03 08:29:06.949781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949756    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-credential-socket" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:06.949781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949739    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-data" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:06.949986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949809    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-certs" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:06.949986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.949972    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istiod-ca-cert" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:06.950321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.950299    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:06.950449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.950324    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:06.950449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.950342    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:06.950449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.950356    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:06.950449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.950371    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:06.951640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.951618    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-kube-api-access-rv2md" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "kube-api-access-rv2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:06.951929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.951911    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-podinfo" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:29:06.952026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.951984    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-envoy" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:06.952026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:06.951988    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-token" pod "f0510f54-3454-46df-b6ee-cbb7b53dfb8c" (UID: "f0510f54-3454-46df-b6ee-cbb7b53dfb8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:07.051194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.051169    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.051194 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.051193    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.051368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.051202    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.051368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.051211    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv2md\" (UniqueName: \"kubernetes.io/projected/f0510f54-3454-46df-b6ee-cbb7b53dfb8c-kube-api-access-rv2md\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.234250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.234224    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"]
Nov 03 08:29:07.237620 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.237597    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-pw7wh"]
Nov 03 08:29:07.829449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.829405    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:29:07.829648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:07.829482    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:29:08.436840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.436813    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:29:08.462534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462455    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921a6364-3e36-4b74-a399-286cccbb26d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a6364-3e36-4b74-a399-286cccbb26d5-istiod-ca-cert\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462501    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-istio-token\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-istio-token\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462535    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-envoy\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462576    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-certs\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462615    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-socket\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462645    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/921a6364-3e36-4b74-a399-286cccbb26d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a6364-3e36-4b74-a399-286cccbb26d5-istio-podinfo\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462669    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-kube-api-access-mh72n\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-kube-api-access-mh72n\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462697    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-credential-socket\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.462772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462726    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-data\") pod \"921a6364-3e36-4b74-a399-286cccbb26d5\" (UID: \"921a6364-3e36-4b74-a399-286cccbb26d5\") "
Nov 03 08:29:08.463179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.462870    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921a6364-3e36-4b74-a399-286cccbb26d5-istiod-ca-cert" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:08.463179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.463102    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921a6364-3e36-4b74-a399-286cccbb26d5-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.463293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.463270    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-credential-socket" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.463354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.463320    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-data" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.463525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.463508    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-certs" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.463693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.463677    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-socket" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.465289 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.465263    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-istio-token" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:08.465444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.465425    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/921a6364-3e36-4b74-a399-286cccbb26d5-istio-podinfo" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:29:08.465574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.465559    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-envoy" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.465920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.465897    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-kube-api-access-mh72n" pod "921a6364-3e36-4b74-a399-286cccbb26d5" (UID: "921a6364-3e36-4b74-a399-286cccbb26d5"). InnerVolumeSpecName "kube-api-access-mh72n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:08.563805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563764    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.563805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563798    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.563805 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563808    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.564047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563819    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.564047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563828    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921a6364-3e36-4b74-a399-286cccbb26d5-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.564047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563837    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh72n\" (UniqueName: \"kubernetes.io/projected/921a6364-3e36-4b74-a399-286cccbb26d5-kube-api-access-mh72n\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.564047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563849    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.564047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.563863    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921a6364-3e36-4b74-a399-286cccbb26d5-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.617428 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.617389    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" path="/var/lib/kubelet/pods/f0510f54-3454-46df-b6ee-cbb7b53dfb8c/volumes"
Nov 03 08:29:08.897591 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.897491    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:08.934162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.934099    6430 generic.go:356] "Generic (PLEG): container finished" podID="921a6364-3e36-4b74-a399-286cccbb26d5" containerID="0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca" exitCode=0
Nov 03 08:29:08.934324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.934162    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" event={"ID":"921a6364-3e36-4b74-a399-286cccbb26d5","Type":"ContainerDied","Data":"0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca"}
Nov 03 08:29:08.934324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.934254    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p" event={"ID":"921a6364-3e36-4b74-a399-286cccbb26d5","Type":"ContainerDied","Data":"c2e9f2b5ecb0c673015be96977aa770879acc3a83cd4174f1490c72d7475d16b"}
Nov 03 08:29:08.934324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.934313    6430 scope.go:122] "RemoveContainer" containerID="0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca"
Nov 03 08:29:08.934517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.934343    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"
Nov 03 08:29:08.949819 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.949342    6430 scope.go:122] "RemoveContainer" containerID="0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca"
Nov 03 08:29:08.950331 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:29:08.950215    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca\": container with ID starting with 0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca not found: ID does not exist" containerID="0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca"
Nov 03 08:29:08.950331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.950259    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca"} err="failed to get container status \"0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca\": rpc error: code = NotFound desc = could not find container \"0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca\": container with ID starting with 0c4504b5de6584f8149002bc34a4037375d793456fec702037b1f1c1ac5e68ca not found: ID does not exist"
Nov 03 08:29:08.973329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.973299    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"]
Nov 03 08:29:08.979379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:08.979351    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-ldm5p"]
Nov 03 08:29:09.014125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:09.014088    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:10.377630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:10.377594    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:10.615404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:10.615309    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" path="/var/lib/kubelet/pods/921a6364-3e36-4b74-a399-286cccbb26d5/volumes"
Nov 03 08:29:12.568570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:12.568533    6430 scope.go:122] "RemoveContainer" containerID="69f30cc0a7b40576c8f797602fcfea54fde22260195584bb664bc9f0098761a4"
Nov 03 08:29:14.477149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:14.477113    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:14.700984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:14.700951    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:14.814505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:14.814423    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:16.092823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:16.092789    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:16.668301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:16.668263    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:21.697192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:21.697159    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:23.904942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.904896    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"]
Nov 03 08:29:23.905410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905347    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fc5c9dff-5472-4849-add5-af54751535e9" containerName="maas-api"
Nov 03 08:29:23.905410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905361    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5c9dff-5472-4849-add5-af54751535e9" containerName="maas-api"
Nov 03 08:29:23.905410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905374    6430 cpu_manager.go:414] "containerMap: removing container" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main"
Nov 03 08:29:23.905410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905379    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main"
Nov 03 08:29:23.905410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905392    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main"
Nov 03 08:29:23.905410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905397    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905415    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5aff661b-468f-4669-a378-bde2349dcab3" containerName="revoke-keys"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905423    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aff661b-468f-4669-a378-bde2349dcab3" containerName="revoke-keys"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905435    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905439    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905450    6430 cpu_manager.go:414] "containerMap: removing container" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905456    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905463    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905467    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905477    6430 cpu_manager.go:414] "containerMap: removing container" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905482    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905544    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08503f1-9bb1-4a0f-b1ab-09f6e8e21838" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905554    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aff661b-468f-4669-a378-bde2349dcab3" containerName="revoke-keys"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905561    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4194f3d-d12c-417b-a715-7fedf4dfd732" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905568    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc94a01-1373-403b-b51c-cc7789cab2d2" containerName="main"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905578    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0510f54-3454-46df-b6ee-cbb7b53dfb8c" containerName="istio-proxy"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905587    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="921a6364-3e36-4b74-a399-286cccbb26d5" containerName="istio-proxy"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905595    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5c9dff-5472-4849-add5-af54751535e9" containerName="maas-api"
Nov 03 08:29:23.905652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.905605    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c5eff3-19ce-4d3e-a475-0ac713d19978" containerName="main"
Nov 03 08:29:23.910973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.910949    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.918770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.918742    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"]
Nov 03 08:29:23.922299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.922282    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.924169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.924145    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"]
Nov 03 08:29:23.934054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.934029    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"]
Nov 03 08:29:23.997513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997474    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.997706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997522    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.997706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997607    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.997706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997639    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.997706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997666    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.997951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997763    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef2358a-4551-4237-bed0-4ad8b79652bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.997951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997836    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.997951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997868    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.997951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997932    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.998159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.997963    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.998159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998050    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.998159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998108    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.998159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998138    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b07e41-a95f-452b-b166-c4532a2daee0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.998327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998181    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.998327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998231    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56dw\" (UniqueName: \"kubernetes.io/projected/6ef2358a-4551-4237-bed0-4ad8b79652bd-kube-api-access-d56dw\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.998327 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998288    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:23.998422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998363    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmqq\" (UniqueName: \"kubernetes.io/projected/f2b07e41-a95f-452b-b166-c4532a2daee0-kube-api-access-7tmqq\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:23.998422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:23.998402    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.099127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099085    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef2358a-4551-4237-bed0-4ad8b79652bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.099320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099145    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.099320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099174    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.099320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099214    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.099498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099342    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.099498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099398    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.099498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099438    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.099498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099485    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b07e41-a95f-452b-b166-c4532a2daee0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.099714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099514    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.099714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099563    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56dw\" (UniqueName: \"kubernetes.io/projected/6ef2358a-4551-4237-bed0-4ad8b79652bd-kube-api-access-d56dw\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.099714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099589    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.099714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099608    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.099714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099632    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.099714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099685    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmqq\" (UniqueName: \"kubernetes.io/projected/f2b07e41-a95f-452b-b166-c4532a2daee0-kube-api-access-7tmqq\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099728    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099737    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099778    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099783    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099806    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099834    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099862    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099864    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099890    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099913    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef2358a-4551-4237-bed0-4ad8b79652bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.099972    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.100651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.100308    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.100651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.100370    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b07e41-a95f-452b-b166-c4532a2daee0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.100651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.100470    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.102674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.102650    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.102674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.102663    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.102813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.102697    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.102813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.102756    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.107221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.107198    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b07e41-a95f-452b-b166-c4532a2daee0-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.107407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.107388    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmqq\" (UniqueName: \"kubernetes.io/projected/f2b07e41-a95f-452b-b166-c4532a2daee0-kube-api-access-7tmqq\") pod \"maas-default-gateway-openshift-default-76f4876667-bchrg\" (UID: \"f2b07e41-a95f-452b-b166-c4532a2daee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.107734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.107715    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56dw\" (UniqueName: \"kubernetes.io/projected/6ef2358a-4551-4237-bed0-4ad8b79652bd-kube-api-access-d56dw\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.108097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.108048    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef2358a-4551-4237-bed0-4ad8b79652bd-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-s92vd\" (UID: \"6ef2358a-4551-4237-bed0-4ad8b79652bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.224278 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.224250    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:24.237681 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.237658    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:24.507468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.507441    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"]
Nov 03 08:29:24.509978 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:29:24.509951    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef2358a_4551_4237_bed0_4ad8b79652bd.slice/crio-1d1e61222c965db411d69fb83cf1c30ee7766d66bced7637a11daaa370fa1bf2 WatchSource:0}: Error finding container 1d1e61222c965db411d69fb83cf1c30ee7766d66bced7637a11daaa370fa1bf2: Status 404 returned error can't find the container with id 1d1e61222c965db411d69fb83cf1c30ee7766d66bced7637a11daaa370fa1bf2
Nov 03 08:29:24.511996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.511968    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:29:24.512095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.512030    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:29:24.512095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.512077    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:29:24.521127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.521101    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"]
Nov 03 08:29:24.523903 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:29:24.523879    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b07e41_a95f_452b_b166_c4532a2daee0.slice/crio-73b6a58416a8a6ac17ef8fc23eba979ee0fbcef3e97b7d3a78ba60316d225a86 WatchSource:0}: Error finding container 73b6a58416a8a6ac17ef8fc23eba979ee0fbcef3e97b7d3a78ba60316d225a86: Status 404 returned error can't find the container with id 73b6a58416a8a6ac17ef8fc23eba979ee0fbcef3e97b7d3a78ba60316d225a86
Nov 03 08:29:24.525920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.525883    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:29:24.526014 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.525966    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:29:24.526104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:24.526014    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:29:25.016086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.014436    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" event={"ID":"f2b07e41-a95f-452b-b166-c4532a2daee0","Type":"ContainerStarted","Data":"98cc9c33ec99a2b73faac952a5b31a0c60b1517d1d3378780c6c03bd3b975666"}
Nov 03 08:29:25.016086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.014503    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" event={"ID":"f2b07e41-a95f-452b-b166-c4532a2daee0","Type":"ContainerStarted","Data":"73b6a58416a8a6ac17ef8fc23eba979ee0fbcef3e97b7d3a78ba60316d225a86"}
Nov 03 08:29:25.023113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.021802    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" event={"ID":"6ef2358a-4551-4237-bed0-4ad8b79652bd","Type":"ContainerStarted","Data":"d77a6568bad4eacb031679d950f69d712c7dbe3da9a35bf465d84a2d11654dab"}
Nov 03 08:29:25.023113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.021865    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" event={"ID":"6ef2358a-4551-4237-bed0-4ad8b79652bd","Type":"ContainerStarted","Data":"1d1e61222c965db411d69fb83cf1c30ee7766d66bced7637a11daaa370fa1bf2"}
Nov 03 08:29:25.043140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.043077    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podStartSLOduration=2.043036293 podStartE2EDuration="2.043036293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:25.041526364 +0000 UTC m=+2114.544361870" watchObservedRunningTime="2026-11-03 08:29:25.043036293 +0000 UTC m=+2114.545871809"
Nov 03 08:29:25.230670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.229462    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:25.237085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.234202    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 08:29:25.237085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.234269    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 08:29:25.239934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.239903    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:25.240326 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.240247    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 08:29:25.240512 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:25.240491    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 08:29:26.226088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:26.225482    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 08:29:26.226088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:26.225547    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 08:29:26.242088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:26.238412    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 08:29:26.242088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:26.238468    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 08:29:27.226540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.226496    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 08:29:27.227228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.227183    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 08:29:27.238169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.238125    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 08:29:27.238415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.238387    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 08:29:27.387350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.387294    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podStartSLOduration=4.3872795270000005 podStartE2EDuration="4.387279527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:25.068416163 +0000 UTC m=+2114.571251678" watchObservedRunningTime="2026-11-03 08:29:27.387279527 +0000 UTC m=+2116.890115040"
Nov 03 08:29:27.388170 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.388134    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"]
Nov 03 08:29:27.391872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.391857    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.394677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.394654    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:29:27.395108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.395087    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:29:27.422407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.422373    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"]
Nov 03 08:29:27.566297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.566195    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/82c7db0c-a0a8-4a62-894c-593bcef7405c-kube-api-access-twswk\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.566297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.566267    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.566531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.566343    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.667392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.667351    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/82c7db0c-a0a8-4a62-894c-593bcef7405c-kube-api-access-twswk\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.667392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.667409    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.667709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.667460    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.668130 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.668107    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.671487 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.671462    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.674491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.674471    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/82c7db0c-a0a8-4a62-894c-593bcef7405c-kube-api-access-twswk\") pod \"maas-api-revoke-sub-85237e45-94fch\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:27.730088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:27.726234    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:28.067183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:28.067145    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"]
Nov 03 08:29:28.070226 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:29:28.070193    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c7db0c_a0a8_4a62_894c_593bcef7405c.slice/crio-22012e80081220045349b6373fb3a3a1002947831b477f7045e5bcf0dd10b6aa WatchSource:0}: Error finding container 22012e80081220045349b6373fb3a3a1002947831b477f7045e5bcf0dd10b6aa: Status 404 returned error can't find the container with id 22012e80081220045349b6373fb3a3a1002947831b477f7045e5bcf0dd10b6aa
Nov 03 08:29:28.228006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:28.227508    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 08:29:28.228006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:28.227576    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 08:29:28.240013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:28.239468    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 08:29:28.240013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:28.239526    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 08:29:28.375469 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:29:28.375431    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:29:29.062370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:29.062334    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch" event={"ID":"82c7db0c-a0a8-4a62-894c-593bcef7405c","Type":"ContainerStarted","Data":"0588a697d35d06e907d07de29414486e73d0f3e1e31010408dcda01ad4e29771"}
Nov 03 08:29:29.062578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:29.062393    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch" event={"ID":"82c7db0c-a0a8-4a62-894c-593bcef7405c","Type":"ContainerStarted","Data":"22012e80081220045349b6373fb3a3a1002947831b477f7045e5bcf0dd10b6aa"}
Nov 03 08:29:29.076669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:29.076615    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch" podStartSLOduration=2.076600654 podStartE2EDuration="2.076600654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:29.076345914 +0000 UTC m=+2118.579181429" watchObservedRunningTime="2026-11-03 08:29:29.076600654 +0000 UTC m=+2118.579436168"
Nov 03 08:29:29.227088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:29.226618    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 08:29:29.227088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:29.226718    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 08:29:29.246086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:29.245252    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 08:29:29.246086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:29.245396    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 08:29:30.068054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:30.068019    6430 generic.go:356] "Generic (PLEG): container finished" podID="82c7db0c-a0a8-4a62-894c-593bcef7405c" containerID="0588a697d35d06e907d07de29414486e73d0f3e1e31010408dcda01ad4e29771" exitCode=0
Nov 03 08:29:30.068287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:30.068118    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch" event={"ID":"82c7db0c-a0a8-4a62-894c-593bcef7405c","Type":"ContainerDied","Data":"0588a697d35d06e907d07de29414486e73d0f3e1e31010408dcda01ad4e29771"}
Nov 03 08:29:30.227086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:30.226478    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 08:29:30.227086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:30.226543    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 08:29:30.246668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:30.246638    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 08:29:30.247181 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:30.247144    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 08:29:31.224785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.224747    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 08:29:31.224931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.224813    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 08:29:31.224931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.224869    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:31.238376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.238350    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 08:29:31.238532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.238394    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 08:29:31.298163 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.298131    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c7db0c-a0a8-4a62-894c-593bcef7405c-kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/82c7db0c-a0a8-4a62-894c-593bcef7405c-kube-api-access-twswk\") pod \"82c7db0c-a0a8-4a62-894c-593bcef7405c\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") "
Nov 03 08:29:31.298670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.298180    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-openshift-ca\") pod \"82c7db0c-a0a8-4a62-894c-593bcef7405c\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") "
Nov 03 08:29:31.298670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.298220    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-xks-ca\") pod \"82c7db0c-a0a8-4a62-894c-593bcef7405c\" (UID: \"82c7db0c-a0a8-4a62-894c-593bcef7405c\") "
Nov 03 08:29:31.298670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.298597    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-openshift-ca" pod "82c7db0c-a0a8-4a62-894c-593bcef7405c" (UID: "82c7db0c-a0a8-4a62-894c-593bcef7405c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:31.300435 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.300403    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c7db0c-a0a8-4a62-894c-593bcef7405c-kube-api-access-twswk" pod "82c7db0c-a0a8-4a62-894c-593bcef7405c" (UID: "82c7db0c-a0a8-4a62-894c-593bcef7405c"). InnerVolumeSpecName "kube-api-access-twswk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:31.300566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.300501    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-xks-ca" pod "82c7db0c-a0a8-4a62-894c-593bcef7405c" (UID: "82c7db0c-a0a8-4a62-894c-593bcef7405c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:29:31.399385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.399298    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/82c7db0c-a0a8-4a62-894c-593bcef7405c-kube-api-access-twswk\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:31.399385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.399342    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:31.399385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:31.399360    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c7db0c-a0a8-4a62-894c-593bcef7405c-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:32.080769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:32.080732    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch" event={"ID":"82c7db0c-a0a8-4a62-894c-593bcef7405c","Type":"ContainerDied","Data":"22012e80081220045349b6373fb3a3a1002947831b477f7045e5bcf0dd10b6aa"}
Nov 03 08:29:32.080992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:32.080781    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22012e80081220045349b6373fb3a3a1002947831b477f7045e5bcf0dd10b6aa"
Nov 03 08:29:32.080992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:32.080777    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"
Nov 03 08:29:32.739000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:32.738962    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"]
Nov 03 08:29:32.739566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:32.739235    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn" podUID="5943db42-3998-4ec5-87f2-77cbda6b5c07" containerName="discovery" containerID="cri-o://991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262" gracePeriod=30
Nov 03 08:29:32.822231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:32.822198    6430 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2z6qn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:8080/ready\": dial tcp 10.133.0.26:8080: connect: connection refused" start-of-body=
Nov 03 08:29:32.822371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:32.822260    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn" podUID="5943db42-3998-4ec5-87f2-77cbda6b5c07" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.26:8080/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Nov 03 08:29:33.024103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.024072    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 08:29:33.087175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.087142    6430 generic.go:356] "Generic (PLEG): container finished" podID="5943db42-3998-4ec5-87f2-77cbda6b5c07" containerID="991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262" exitCode=0
Nov 03 08:29:33.087376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.087188    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"
Nov 03 08:29:33.087376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.087219    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn" event={"ID":"5943db42-3998-4ec5-87f2-77cbda6b5c07","Type":"ContainerDied","Data":"991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262"}
Nov 03 08:29:33.087376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.087267    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn" event={"ID":"5943db42-3998-4ec5-87f2-77cbda6b5c07","Type":"ContainerDied","Data":"8b8079b4374c0109dc047bc0ac8d3c607cf04df2e9668f54db9440a6ed538392"}
Nov 03 08:29:33.087376 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.087292    6430 scope.go:122] "RemoveContainer" containerID="991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262"
Nov 03 08:29:33.108240 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.108217    6430 scope.go:122] "RemoveContainer" containerID="991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262"
Nov 03 08:29:33.108576 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:29:33.108549    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262\": container with ID starting with 991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262 not found: ID does not exist" containerID="991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262"
Nov 03 08:29:33.108642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.108592    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262"} err="failed to get container status \"991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262\": rpc error: code = NotFound desc = could not find container \"991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262\": container with ID starting with 991fc495e517e219f0be6e859312bb0b1e2dd2f46a275d05b842ddbab047b262 not found: ID does not exist"
Nov 03 08:29:33.113806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.113788    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5943db42-3998-4ec5-87f2-77cbda6b5c07-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5943db42-3998-4ec5-87f2-77cbda6b5c07-local-certs\") pod \"5943db42-3998-4ec5-87f2-77cbda6b5c07\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") "
Nov 03 08:29:33.113888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.113835    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-ca-configmap\") pod \"5943db42-3998-4ec5-87f2-77cbda6b5c07\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") "
Nov 03 08:29:33.113888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.113875    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-cacerts\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-cacerts\") pod \"5943db42-3998-4ec5-87f2-77cbda6b5c07\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") "
Nov 03 08:29:33.114001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.113901    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-kube-api-access-g26fd\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-kube-api-access-g26fd\") pod \"5943db42-3998-4ec5-87f2-77cbda6b5c07\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") "
Nov 03 08:29:33.114001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.113930    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-token\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-token\") pod \"5943db42-3998-4ec5-87f2-77cbda6b5c07\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") "
Nov 03 08:29:33.114001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.113979    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-kubeconfig\") pod \"5943db42-3998-4ec5-87f2-77cbda6b5c07\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") "
Nov 03 08:29:33.114185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.114115    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-dns-cert\") pod \"5943db42-3998-4ec5-87f2-77cbda6b5c07\" (UID: \"5943db42-3998-4ec5-87f2-77cbda6b5c07\") "
Nov 03 08:29:33.114533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.114351    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-ca-configmap" pod "5943db42-3998-4ec5-87f2-77cbda6b5c07" (UID: "5943db42-3998-4ec5-87f2-77cbda6b5c07"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:33.114533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.114502    6430 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-ca-configmap\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:33.116896 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.116848    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-cacerts" pod "5943db42-3998-4ec5-87f2-77cbda6b5c07" (UID: "5943db42-3998-4ec5-87f2-77cbda6b5c07"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:29:33.117303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.117276    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-kubeconfig" pod "5943db42-3998-4ec5-87f2-77cbda6b5c07" (UID: "5943db42-3998-4ec5-87f2-77cbda6b5c07"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:29:33.117423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.117358    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-kube-api-access-g26fd" pod "5943db42-3998-4ec5-87f2-77cbda6b5c07" (UID: "5943db42-3998-4ec5-87f2-77cbda6b5c07"). InnerVolumeSpecName "kube-api-access-g26fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:33.118845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.118343    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-dns-cert" pod "5943db42-3998-4ec5-87f2-77cbda6b5c07" (UID: "5943db42-3998-4ec5-87f2-77cbda6b5c07"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:29:33.118845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.118515    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-token" pod "5943db42-3998-4ec5-87f2-77cbda6b5c07" (UID: "5943db42-3998-4ec5-87f2-77cbda6b5c07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:33.119156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.119136    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5943db42-3998-4ec5-87f2-77cbda6b5c07-local-certs" pod "5943db42-3998-4ec5-87f2-77cbda6b5c07" (UID: "5943db42-3998-4ec5-87f2-77cbda6b5c07"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:33.216198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.216155    6430 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-cacerts\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:33.216198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.216193    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26fd\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-kube-api-access-g26fd\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:33.216472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.216209    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:33.216472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.216223    6430 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-kubeconfig\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:33.216472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.216236    6430 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5943db42-3998-4ec5-87f2-77cbda6b5c07-istio-csr-dns-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:33.216472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.216249    6430 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5943db42-3998-4ec5-87f2-77cbda6b5c07-local-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:33.226188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.225974    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:33.226188 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.226054    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:33.239477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.239436    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:33.239698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.239513    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:33.413254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.413216    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"]
Nov 03 08:29:33.418599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:33.418569    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z6qn"]
Nov 03 08:29:34.267225 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:34.267192    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:34.267668 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:34.267260    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:34.280183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:34.280157    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:34.280319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:34.280197    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:34.623821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:34.623368    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5943db42-3998-4ec5-87f2-77cbda6b5c07" path="/var/lib/kubelet/pods/5943db42-3998-4ec5-87f2-77cbda6b5c07/volumes"
Nov 03 08:29:35.308275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.308223    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-s92vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:35.308829 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.308312    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd" podUID="6ef2358a-4551-4237-bed0-4ad8b79652bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:35.313667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.313638    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:35.314810 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.314786    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:35.315556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.315534    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-s92vd"
Nov 03 08:29:35.322232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.322201    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bchrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:35.322335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.322254    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg" podUID="f2b07e41-a95f-452b-b166-c4532a2daee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:35.433818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.433790    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:35.434124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.434098    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:35.434779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:35.434761    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bchrg"
Nov 03 08:29:38.847274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:38.847229    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:40.649139 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.649081    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:29:40.764910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.764878    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-799454fcf4-2rncm"]
Nov 03 08:29:40.765235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.765224    6430 cpu_manager.go:414] "containerMap: removing container" podUID="82c7db0c-a0a8-4a62-894c-593bcef7405c" containerName="revoke-keys"
Nov 03 08:29:40.765291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.765237    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c7db0c-a0a8-4a62-894c-593bcef7405c" containerName="revoke-keys"
Nov 03 08:29:40.765291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.765265    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5943db42-3998-4ec5-87f2-77cbda6b5c07" containerName="discovery"
Nov 03 08:29:40.765291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.765272    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5943db42-3998-4ec5-87f2-77cbda6b5c07" containerName="discovery"
Nov 03 08:29:40.765393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.765324    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c7db0c-a0a8-4a62-894c-593bcef7405c" containerName="revoke-keys"
Nov 03 08:29:40.765393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.765339    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5943db42-3998-4ec5-87f2-77cbda6b5c07" containerName="discovery"
Nov 03 08:29:40.768397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.768379    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.772116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.772096    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k97gv\""
Nov 03 08:29:40.772210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.772135    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 08:29:40.774910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.774772    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 08:29:40.779531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.779513    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 08:29:40.780669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.780646    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799454fcf4-2rncm"]
Nov 03 08:29:40.894950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.894907    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/25c36014-1407-4536-ad79-b3b4e16be4c0-kube-api-access-7f92g\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.895125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.895029    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25c36014-1407-4536-ad79-b3b4e16be4c0-webhook-cert\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.895213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.895144    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c36014-1407-4536-ad79-b3b4e16be4c0-metrics-certs\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.995823 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.995797    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25c36014-1407-4536-ad79-b3b4e16be4c0-webhook-cert\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.995937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.995841    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c36014-1407-4536-ad79-b3b4e16be4c0-metrics-certs\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.995937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.995874    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/25c36014-1407-4536-ad79-b3b4e16be4c0-kube-api-access-7f92g\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.998468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.998444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c36014-1407-4536-ad79-b3b4e16be4c0-metrics-certs\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:40.998580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:40.998514    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25c36014-1407-4536-ad79-b3b4e16be4c0-webhook-cert\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:41.003570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:41.003552    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/25c36014-1407-4536-ad79-b3b4e16be4c0-kube-api-access-7f92g\") pod \"maas-controller-799454fcf4-2rncm\" (UID: \"25c36014-1407-4536-ad79-b3b4e16be4c0\") " pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:41.081254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:41.081231    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:41.354836 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:41.354749    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799454fcf4-2rncm"]
Nov 03 08:29:41.358024 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:29:41.357996    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c36014_1407_4536_ad79_b3b4e16be4c0.slice/crio-f731141860d175017cd616081889031a0a9a114124ac72258adaea46401ea00c WatchSource:0}: Error finding container f731141860d175017cd616081889031a0a9a114124ac72258adaea46401ea00c: Status 404 returned error can't find the container with id f731141860d175017cd616081889031a0a9a114124ac72258adaea46401ea00c
Nov 03 08:29:41.359772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:41.359749    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:29:42.128421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:42.128378    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-2rncm" event={"ID":"25c36014-1407-4536-ad79-b3b4e16be4c0","Type":"ContainerStarted","Data":"f731141860d175017cd616081889031a0a9a114124ac72258adaea46401ea00c"}
Nov 03 08:29:43.134518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:43.134479    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-2rncm" event={"ID":"25c36014-1407-4536-ad79-b3b4e16be4c0","Type":"ContainerStarted","Data":"5950f0ba1638bcf6a1fd27a9c758971bc38794bbd180ed253522f6660cbdedde"}
Nov 03 08:29:43.134950 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:43.134583    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:43.150083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:43.149991    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-799454fcf4-2rncm" podStartSLOduration=1.529342526 podStartE2EDuration="3.149978203s" totalImagesPullingTime="1.620635677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:43.149186833 +0000 UTC m=+2132.652022348" watchObservedRunningTime="2026-11-03 08:29:43.149978203 +0000 UTC m=+2132.652813716"
Nov 03 08:29:54.140842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:29:54.140813    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-799454fcf4-2rncm"
Nov 03 08:29:58.374180 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:29:58.374133    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:30:00.136715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.136674    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"]
Nov 03 08:30:00.140509 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.140486    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.161049 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.161010    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"]
Nov 03 08:30:00.163990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.163952    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwpw\" (UniqueName: \"kubernetes.io/projected/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-kube-api-access-2hwpw\") pod \"maas-api-key-cleanup-29894910-sz7wc\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.164177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.164089    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894910-sz7wc\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.190099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.190044    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:30:00.265506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.265463    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwpw\" (UniqueName: \"kubernetes.io/projected/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-kube-api-access-2hwpw\") pod \"maas-api-key-cleanup-29894910-sz7wc\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.265689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.265522    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894910-sz7wc\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.266231 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.266211    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894910-sz7wc\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.279336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.279308    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwpw\" (UniqueName: \"kubernetes.io/projected/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-kube-api-access-2hwpw\") pod \"maas-api-key-cleanup-29894910-sz7wc\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.454798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.454765    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:00.731430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:00.731352    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"]
Nov 03 08:30:00.734686 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:30:00.734656    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189b63f6_3d43_4b28_bdeb_c5fe6f1dba70.slice/crio-dbd6571d861fa064b8e24756b14b110d97da82e16bb85fc61c770a6bd7c17d82 WatchSource:0}: Error finding container dbd6571d861fa064b8e24756b14b110d97da82e16bb85fc61c770a6bd7c17d82: Status 404 returned error can't find the container with id dbd6571d861fa064b8e24756b14b110d97da82e16bb85fc61c770a6bd7c17d82
Nov 03 08:30:01.216051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:01.216007    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" event={"ID":"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70","Type":"ContainerStarted","Data":"5b4a4905e4eae29a545e4da91795a4a5f7187d75eee26e3827f6c9c5b3ef169c"}
Nov 03 08:30:01.216508 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:01.216084    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" event={"ID":"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70","Type":"ContainerStarted","Data":"dbd6571d861fa064b8e24756b14b110d97da82e16bb85fc61c770a6bd7c17d82"}
Nov 03 08:30:01.230126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:01.230043    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" podStartSLOduration=1.230030612 podStartE2EDuration="1.230030612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:30:01.228992386 +0000 UTC m=+2150.731827900" watchObservedRunningTime="2026-11-03 08:30:01.230030612 +0000 UTC m=+2150.732866126"
Nov 03 08:30:02.228169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:02.228135    6430 generic.go:356] "Generic (PLEG): container finished" podID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerID="5b4a4905e4eae29a545e4da91795a4a5f7187d75eee26e3827f6c9c5b3ef169c" exitCode=60
Nov 03 08:30:02.228565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:02.228196    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" event={"ID":"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70","Type":"ContainerDied","Data":"5b4a4905e4eae29a545e4da91795a4a5f7187d75eee26e3827f6c9c5b3ef169c"}
Nov 03 08:30:02.228565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:02.228445    6430 scope.go:122] "RemoveContainer" containerID="5b4a4905e4eae29a545e4da91795a4a5f7187d75eee26e3827f6c9c5b3ef169c"
Nov 03 08:30:03.234260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:03.234223    6430 generic.go:356] "Generic (PLEG): container finished" podID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerID="60676da1b945d53f5616fe2e20b366e9cc1f2fed8450fda0015ae555b4a429fe" exitCode=60
Nov 03 08:30:03.234734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:03.234273    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" event={"ID":"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70","Type":"ContainerDied","Data":"60676da1b945d53f5616fe2e20b366e9cc1f2fed8450fda0015ae555b4a429fe"}
Nov 03 08:30:03.234734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:03.234325    6430 scope.go:122] "RemoveContainer" containerID="5b4a4905e4eae29a545e4da91795a4a5f7187d75eee26e3827f6c9c5b3ef169c"
Nov 03 08:30:03.234734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:03.234518    6430 scope.go:122] "RemoveContainer" containerID="60676da1b945d53f5616fe2e20b366e9cc1f2fed8450fda0015ae555b4a429fe"
Nov 03 08:30:03.234880 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:30:03.234741    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894910-sz7wc_odh-ai-gateway-infra(189b63f6-3d43-4b28-bdeb-c5fe6f1dba70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70"
Nov 03 08:30:04.239621 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:04.239586    6430 scope.go:122] "RemoveContainer" containerID="60676da1b945d53f5616fe2e20b366e9cc1f2fed8450fda0015ae555b4a429fe"
Nov 03 08:30:04.239998 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:30:04.239779    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894910-sz7wc_odh-ai-gateway-infra(189b63f6-3d43-4b28-bdeb-c5fe6f1dba70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70"
Nov 03 08:30:14.528039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:14.527994    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:30:14.610839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:14.610808    6430 scope.go:122] "RemoveContainer" containerID="60676da1b945d53f5616fe2e20b366e9cc1f2fed8450fda0015ae555b4a429fe"
Nov 03 08:30:15.288880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:15.288847    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" event={"ID":"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70","Type":"ContainerStarted","Data":"1b1d10187c855f8512c83b8d6e47ae88581a5558e8dde4048fb6d4e836de95d4"}
Nov 03 08:30:16.296373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:16.296338    6430 generic.go:356] "Generic (PLEG): container finished" podID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerID="1b1d10187c855f8512c83b8d6e47ae88581a5558e8dde4048fb6d4e836de95d4" exitCode=60
Nov 03 08:30:16.296885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:16.296414    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" event={"ID":"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70","Type":"ContainerDied","Data":"1b1d10187c855f8512c83b8d6e47ae88581a5558e8dde4048fb6d4e836de95d4"}
Nov 03 08:30:16.296885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:16.296494    6430 scope.go:122] "RemoveContainer" containerID="60676da1b945d53f5616fe2e20b366e9cc1f2fed8450fda0015ae555b4a429fe"
Nov 03 08:30:16.296885 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:16.296826    6430 scope.go:122] "RemoveContainer" containerID="1b1d10187c855f8512c83b8d6e47ae88581a5558e8dde4048fb6d4e836de95d4"
Nov 03 08:30:16.297156 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:30:16.297128    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894910-sz7wc_odh-ai-gateway-infra(189b63f6-3d43-4b28-bdeb-c5fe6f1dba70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70"
Nov 03 08:30:16.312837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:16.312805    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"]
Nov 03 08:30:17.630082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:17.627327    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:17.775088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:17.768218    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-kube-api-access-2hwpw\" (UniqueName: \"kubernetes.io/projected/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-kube-api-access-2hwpw\") pod \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") "
Nov 03 08:30:17.775088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:17.768381    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-maas-api-service-ca\") pod \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\" (UID: \"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70\") "
Nov 03 08:30:17.775088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:17.768775    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-maas-api-service-ca" pod "189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" (UID: "189b63f6-3d43-4b28-bdeb-c5fe6f1dba70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:30:17.775088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:17.769043    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-maas-api-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:17.775088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:17.770509    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-kube-api-access-2hwpw" pod "189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" (UID: "189b63f6-3d43-4b28-bdeb-c5fe6f1dba70"). InnerVolumeSpecName "kube-api-access-2hwpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:30:17.874288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:17.870251    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwpw\" (UniqueName: \"kubernetes.io/projected/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70-kube-api-access-2hwpw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:18.350607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:18.350482    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc" event={"ID":"189b63f6-3d43-4b28-bdeb-c5fe6f1dba70","Type":"ContainerDied","Data":"dbd6571d861fa064b8e24756b14b110d97da82e16bb85fc61c770a6bd7c17d82"}
Nov 03 08:30:18.350930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:18.350577    6430 scope.go:122] "RemoveContainer" containerID="1b1d10187c855f8512c83b8d6e47ae88581a5558e8dde4048fb6d4e836de95d4"
Nov 03 08:30:18.351213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:18.351181    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"
Nov 03 08:30:18.383740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:18.383690    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"]
Nov 03 08:30:18.385583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:18.385556    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894910-sz7wc"]
Nov 03 08:30:18.615634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:18.615538    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" path="/var/lib/kubelet/pods/189b63f6-3d43-4b28-bdeb-c5fe6f1dba70/volumes"
Nov 03 08:30:22.950724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.950642    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"]
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951001    6430 cpu_manager.go:414] "containerMap: removing container" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951011    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951026    6430 cpu_manager.go:414] "containerMap: removing container" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951031    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951038    6430 cpu_manager.go:414] "containerMap: removing container" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951045    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951127    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.951134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.951136    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:22.955489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.955472    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:22.962718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:22.962693    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"]
Nov 03 08:30:23.004750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.004719    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:30:23.004990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.004844    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:30:23.014176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.014148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.014351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.014225    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwjf\" (UniqueName: \"kubernetes.io/projected/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-kube-api-access-mdwjf\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.014351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.014262    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.114845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.114807    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.115053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.114896    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.115053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.114916    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwjf\" (UniqueName: \"kubernetes.io/projected/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-kube-api-access-mdwjf\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.115584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.115556    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.117538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.117505    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.124119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.124099    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwjf\" (UniqueName: \"kubernetes.io/projected/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-kube-api-access-mdwjf\") pod \"maas-api-revoke-sub-c1e271a9-nn9gr\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.270308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.270210    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:23.541618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:23.541532    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"]
Nov 03 08:30:23.545690 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:30:23.545659    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57143e0_6ac9_4fde_855e_8bb7e43fd1e2.slice/crio-2d3ee1c6df1e5caef9ae36430fe53d839ee85b445be736927060f8ebf79f3d5d WatchSource:0}: Error finding container 2d3ee1c6df1e5caef9ae36430fe53d839ee85b445be736927060f8ebf79f3d5d: Status 404 returned error can't find the container with id 2d3ee1c6df1e5caef9ae36430fe53d839ee85b445be736927060f8ebf79f3d5d
Nov 03 08:30:24.381596 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:24.381561    6430 generic.go:356] "Generic (PLEG): container finished" podID="e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" containerID="a10267f84b88575e1f6a043a87b0401d222370c208fd42cdb534bdef2ec9f128" exitCode=0
Nov 03 08:30:24.381984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:24.381645    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr" event={"ID":"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2","Type":"ContainerDied","Data":"a10267f84b88575e1f6a043a87b0401d222370c208fd42cdb534bdef2ec9f128"}
Nov 03 08:30:24.381984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:24.381702    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr" event={"ID":"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2","Type":"ContainerStarted","Data":"2d3ee1c6df1e5caef9ae36430fe53d839ee85b445be736927060f8ebf79f3d5d"}
Nov 03 08:30:25.517672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.517648    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:25.532744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.532549    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-xks-ca\") pod \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") "
Nov 03 08:30:25.532744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.532613    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-kube-api-access-mdwjf\" (UniqueName: \"kubernetes.io/projected/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-kube-api-access-mdwjf\") pod \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") "
Nov 03 08:30:25.532744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.532695    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-openshift-ca\") pod \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\" (UID: \"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2\") "
Nov 03 08:30:25.533280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.533253    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-openshift-ca" pod "e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" (UID: "e57143e0-6ac9-4fde-855e-8bb7e43fd1e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:30:25.535202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.535174    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-xks-ca" pod "e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" (UID: "e57143e0-6ac9-4fde-855e-8bb7e43fd1e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:30:25.535489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.535463    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-kube-api-access-mdwjf" pod "e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" (UID: "e57143e0-6ac9-4fde-855e-8bb7e43fd1e2"). InnerVolumeSpecName "kube-api-access-mdwjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:30:25.633653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.633613    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwjf\" (UniqueName: \"kubernetes.io/projected/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-kube-api-access-mdwjf\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:25.633653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.633648    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:25.633653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:25.633661    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:26.390421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:26.390390    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"
Nov 03 08:30:26.390421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:26.390410    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr" event={"ID":"e57143e0-6ac9-4fde-855e-8bb7e43fd1e2","Type":"ContainerDied","Data":"2d3ee1c6df1e5caef9ae36430fe53d839ee85b445be736927060f8ebf79f3d5d"}
Nov 03 08:30:26.390661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:26.390465    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d3ee1c6df1e5caef9ae36430fe53d839ee85b445be736927060f8ebf79f3d5d"
Nov 03 08:30:28.374614 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:30:28.374576    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:30:41.538644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.538599    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"]
Nov 03 08:30:41.539089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.538993    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" containerName="revoke-keys"
Nov 03 08:30:41.539089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.539005    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" containerName="revoke-keys"
Nov 03 08:30:41.539200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.539103    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="189b63f6-3d43-4b28-bdeb-c5fe6f1dba70" containerName="cleanup"
Nov 03 08:30:41.539200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.539120    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" containerName="revoke-keys"
Nov 03 08:30:41.542125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.542108    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.544674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.544651    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:30:41.545106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.545086    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:30:41.550174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.550147    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"]
Nov 03 08:30:41.589172 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.589137    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrnq\" (UniqueName: \"kubernetes.io/projected/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-kube-api-access-scrnq\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.589335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.589206    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.589419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.589335    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.689993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.689953    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.690185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.690028    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.690185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.690124    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrnq\" (UniqueName: \"kubernetes.io/projected/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-kube-api-access-scrnq\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.690772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.690751    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.692607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.692588    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.699111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.699087    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrnq\" (UniqueName: \"kubernetes.io/projected/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-kube-api-access-scrnq\") pod \"maas-api-revoke-sub-2e1f5b5f-5ml2s\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:41.860171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:41.860080    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:42.131281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:42.131206    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"]
Nov 03 08:30:42.134638 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:30:42.134609    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9084f7d_a9fe_4619_a561_71b4f5cd5b5f.slice/crio-b46dfcfc5ccfd0422d66b7e2e57f22bcb1209ab1faf38d2df75704a96bb25e04 WatchSource:0}: Error finding container b46dfcfc5ccfd0422d66b7e2e57f22bcb1209ab1faf38d2df75704a96bb25e04: Status 404 returned error can't find the container with id b46dfcfc5ccfd0422d66b7e2e57f22bcb1209ab1faf38d2df75704a96bb25e04
Nov 03 08:30:42.467369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:42.467334    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s" event={"ID":"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f","Type":"ContainerStarted","Data":"2d2e79a7c8889eb4fe052026f196adfe28065eb1ce0765bba036bfbc5bf9b80d"}
Nov 03 08:30:42.467543 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:42.467386    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s" event={"ID":"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f","Type":"ContainerStarted","Data":"b46dfcfc5ccfd0422d66b7e2e57f22bcb1209ab1faf38d2df75704a96bb25e04"}
Nov 03 08:30:42.482217 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:42.482174    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s" podStartSLOduration=1.482161716 podStartE2EDuration="1.482161716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:30:42.481989612 +0000 UTC m=+2191.984825126" watchObservedRunningTime="2026-11-03 08:30:42.482161716 +0000 UTC m=+2191.984997229"
Nov 03 08:30:43.473132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:43.473097    6430 generic.go:356] "Generic (PLEG): container finished" podID="f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" containerID="2d2e79a7c8889eb4fe052026f196adfe28065eb1ce0765bba036bfbc5bf9b80d" exitCode=0
Nov 03 08:30:43.473535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:43.473172    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s" event={"ID":"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f","Type":"ContainerDied","Data":"2d2e79a7c8889eb4fe052026f196adfe28065eb1ce0765bba036bfbc5bf9b80d"}
Nov 03 08:30:44.606440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.606415    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:44.720502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.720467    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-openshift-ca\") pod \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") "
Nov 03 08:30:44.720679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.720528    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-kube-api-access-scrnq\" (UniqueName: \"kubernetes.io/projected/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-kube-api-access-scrnq\") pod \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") "
Nov 03 08:30:44.720679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.720618    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-xks-ca\") pod \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\" (UID: \"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f\") "
Nov 03 08:30:44.720895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.720869    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-openshift-ca" pod "f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" (UID: "f9084f7d-a9fe-4619-a561-71b4f5cd5b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:30:44.722820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.722791    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-kube-api-access-scrnq" pod "f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" (UID: "f9084f7d-a9fe-4619-a561-71b4f5cd5b5f"). InnerVolumeSpecName "kube-api-access-scrnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:30:44.722931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.722838    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-xks-ca" pod "f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" (UID: "f9084f7d-a9fe-4619-a561-71b4f5cd5b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:30:44.821880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.821816    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scrnq\" (UniqueName: \"kubernetes.io/projected/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-kube-api-access-scrnq\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:44.821880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.821843    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:44.821880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:44.821853    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:45.483840 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:45.483804    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s" event={"ID":"f9084f7d-a9fe-4619-a561-71b4f5cd5b5f","Type":"ContainerDied","Data":"b46dfcfc5ccfd0422d66b7e2e57f22bcb1209ab1faf38d2df75704a96bb25e04"}
Nov 03 08:30:45.484108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:45.483848    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b46dfcfc5ccfd0422d66b7e2e57f22bcb1209ab1faf38d2df75704a96bb25e04"
Nov 03 08:30:45.484108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:45.483812    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"
Nov 03 08:30:52.489600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.489564    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"]
Nov 03 08:30:52.489983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.489899    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" containerName="revoke-keys"
Nov 03 08:30:52.489983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.489909    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" containerName="revoke-keys"
Nov 03 08:30:52.490097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.489986    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" containerName="revoke-keys"
Nov 03 08:30:52.493310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.493289    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.496174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.496151    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:30:52.496641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.496610    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:30:52.506198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.506175    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"]
Nov 03 08:30:52.586744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.586712    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.586910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.586783    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skx9m\" (UniqueName: \"kubernetes.io/projected/c1c29096-7b1f-4132-b9b0-12eca3efa44f-kube-api-access-skx9m\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.586910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.586858    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.687616 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.687575    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.687774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.687660    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.687774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.687706    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9m\" (UniqueName: \"kubernetes.io/projected/c1c29096-7b1f-4132-b9b0-12eca3efa44f-kube-api-access-skx9m\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.688396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.688369    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.690158 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.690135    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.694865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.694832    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skx9m\" (UniqueName: \"kubernetes.io/projected/c1c29096-7b1f-4132-b9b0-12eca3efa44f-kube-api-access-skx9m\") pod \"maas-api-revoke-sub-ec843018-w9jkj\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:52.807421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:52.807339    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:53.093677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:53.093600    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"]
Nov 03 08:30:53.097496 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:30:53.097470    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c29096_7b1f_4132_b9b0_12eca3efa44f.slice/crio-9fce789ce9cbc61978a356a87068720e505c6ce28ffd86077919aef5aa40c6dd WatchSource:0}: Error finding container 9fce789ce9cbc61978a356a87068720e505c6ce28ffd86077919aef5aa40c6dd: Status 404 returned error can't find the container with id 9fce789ce9cbc61978a356a87068720e505c6ce28ffd86077919aef5aa40c6dd
Nov 03 08:30:53.524963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:53.524921    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj" event={"ID":"c1c29096-7b1f-4132-b9b0-12eca3efa44f","Type":"ContainerStarted","Data":"22dc61e2fc52e405d2384f8d388c2ca6b9da962ad27dddeee4b2832b83538d87"}
Nov 03 08:30:53.525398 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:53.525007    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj" event={"ID":"c1c29096-7b1f-4132-b9b0-12eca3efa44f","Type":"ContainerStarted","Data":"9fce789ce9cbc61978a356a87068720e505c6ce28ffd86077919aef5aa40c6dd"}
Nov 03 08:30:53.540793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:53.540741    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj" podStartSLOduration=1.540724923 podStartE2EDuration="1.540724923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:30:53.538988222 +0000 UTC m=+2203.041823746" watchObservedRunningTime="2026-11-03 08:30:53.540724923 +0000 UTC m=+2203.043560435"
Nov 03 08:30:54.533245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:54.533208    6430 generic.go:356] "Generic (PLEG): container finished" podID="c1c29096-7b1f-4132-b9b0-12eca3efa44f" containerID="22dc61e2fc52e405d2384f8d388c2ca6b9da962ad27dddeee4b2832b83538d87" exitCode=0
Nov 03 08:30:54.533785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:54.533284    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj" event={"ID":"c1c29096-7b1f-4132-b9b0-12eca3efa44f","Type":"ContainerDied","Data":"22dc61e2fc52e405d2384f8d388c2ca6b9da962ad27dddeee4b2832b83538d87"}
Nov 03 08:30:55.893013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:55.892988    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:56.038006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.034346    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c29096-7b1f-4132-b9b0-12eca3efa44f-kube-api-access-skx9m\" (UniqueName: \"kubernetes.io/projected/c1c29096-7b1f-4132-b9b0-12eca3efa44f-kube-api-access-skx9m\") pod \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") "
Nov 03 08:30:56.038006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.034407    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-openshift-ca\") pod \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") "
Nov 03 08:30:56.038006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.034445    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-xks-ca\") pod \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\" (UID: \"c1c29096-7b1f-4132-b9b0-12eca3efa44f\") "
Nov 03 08:30:56.038006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.035718    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-openshift-ca" pod "c1c29096-7b1f-4132-b9b0-12eca3efa44f" (UID: "c1c29096-7b1f-4132-b9b0-12eca3efa44f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:30:56.038917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.038876    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c29096-7b1f-4132-b9b0-12eca3efa44f-kube-api-access-skx9m" pod "c1c29096-7b1f-4132-b9b0-12eca3efa44f" (UID: "c1c29096-7b1f-4132-b9b0-12eca3efa44f"). InnerVolumeSpecName "kube-api-access-skx9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:30:56.072109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.071533    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-xks-ca" pod "c1c29096-7b1f-4132-b9b0-12eca3efa44f" (UID: "c1c29096-7b1f-4132-b9b0-12eca3efa44f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:30:56.140657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.140539    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:56.140657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.140572    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c29096-7b1f-4132-b9b0-12eca3efa44f-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:56.140657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.140588    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skx9m\" (UniqueName: \"kubernetes.io/projected/c1c29096-7b1f-4132-b9b0-12eca3efa44f-kube-api-access-skx9m\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:30:56.568845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.568806    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj" event={"ID":"c1c29096-7b1f-4132-b9b0-12eca3efa44f","Type":"ContainerDied","Data":"9fce789ce9cbc61978a356a87068720e505c6ce28ffd86077919aef5aa40c6dd"}
Nov 03 08:30:56.569140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.569118    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fce789ce9cbc61978a356a87068720e505c6ce28ffd86077919aef5aa40c6dd"
Nov 03 08:30:56.569421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:30:56.569369    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"
Nov 03 08:30:58.373958 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:30:58.373914    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:31:14.575226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:14.575192    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:31:18.569098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.569039    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"]
Nov 03 08:31:18.569754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.569446    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c1c29096-7b1f-4132-b9b0-12eca3efa44f" containerName="revoke-keys"
Nov 03 08:31:18.569754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.569466    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c29096-7b1f-4132-b9b0-12eca3efa44f" containerName="revoke-keys"
Nov 03 08:31:18.569754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.569556    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c29096-7b1f-4132-b9b0-12eca3efa44f" containerName="revoke-keys"
Nov 03 08:31:18.573852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.573830    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.581379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.580943    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"]
Nov 03 08:31:18.582313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.582279    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:31:18.582438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.582341    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:31:18.669903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.669869    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.670095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.669912    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlm9\" (UniqueName: \"kubernetes.io/projected/91a89702-510f-436c-8443-b1eba9764b6d-kube-api-access-bjlm9\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.670095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.669941    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.771151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.771112    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.771338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.771161    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlm9\" (UniqueName: \"kubernetes.io/projected/91a89702-510f-436c-8443-b1eba9764b6d-kube-api-access-bjlm9\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.771338 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.771193    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.771941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.771857    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.773592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.773567    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.778793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.778773    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlm9\" (UniqueName: \"kubernetes.io/projected/91a89702-510f-436c-8443-b1eba9764b6d-kube-api-access-bjlm9\") pod \"maas-api-revoke-sub-046e1343-c2kgf\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:18.889116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:18.889023    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:19.161557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:19.161480    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"]
Nov 03 08:31:19.164743 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:31:19.164713    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a89702_510f_436c_8443_b1eba9764b6d.slice/crio-27dd63bad915bcbddecdda301a4a8ea370531fa161b92ab44eed1c4b01a54f0d WatchSource:0}: Error finding container 27dd63bad915bcbddecdda301a4a8ea370531fa161b92ab44eed1c4b01a54f0d: Status 404 returned error can't find the container with id 27dd63bad915bcbddecdda301a4a8ea370531fa161b92ab44eed1c4b01a54f0d
Nov 03 08:31:19.724989 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:19.724945    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf" event={"ID":"91a89702-510f-436c-8443-b1eba9764b6d","Type":"ContainerStarted","Data":"07e075667bee3f68ff921a4e202f662227b5a644d9af83b9fb3d6da8b92ae752"}
Nov 03 08:31:19.725490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:19.725031    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf" event={"ID":"91a89702-510f-436c-8443-b1eba9764b6d","Type":"ContainerStarted","Data":"27dd63bad915bcbddecdda301a4a8ea370531fa161b92ab44eed1c4b01a54f0d"}
Nov 03 08:31:19.738992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:19.738946    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf" podStartSLOduration=1.738934499 podStartE2EDuration="1.738934499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:31:19.73827445 +0000 UTC m=+2229.241109965" watchObservedRunningTime="2026-11-03 08:31:19.738934499 +0000 UTC m=+2229.241770058"
Nov 03 08:31:20.730528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:20.730494    6430 generic.go:356] "Generic (PLEG): container finished" podID="91a89702-510f-436c-8443-b1eba9764b6d" containerID="07e075667bee3f68ff921a4e202f662227b5a644d9af83b9fb3d6da8b92ae752" exitCode=0
Nov 03 08:31:20.730932 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:20.730534    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf" event={"ID":"91a89702-510f-436c-8443-b1eba9764b6d","Type":"ContainerDied","Data":"07e075667bee3f68ff921a4e202f662227b5a644d9af83b9fb3d6da8b92ae752"}
Nov 03 08:31:21.858741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.858714    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:21.897572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.897535    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-openshift-ca\") pod \"91a89702-510f-436c-8443-b1eba9764b6d\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") "
Nov 03 08:31:21.897722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.897592    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-xks-ca\") pod \"91a89702-510f-436c-8443-b1eba9764b6d\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") "
Nov 03 08:31:21.897768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.897731    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a89702-510f-436c-8443-b1eba9764b6d-kube-api-access-bjlm9\" (UniqueName: \"kubernetes.io/projected/91a89702-510f-436c-8443-b1eba9764b6d-kube-api-access-bjlm9\") pod \"91a89702-510f-436c-8443-b1eba9764b6d\" (UID: \"91a89702-510f-436c-8443-b1eba9764b6d\") "
Nov 03 08:31:21.897852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.897831    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-openshift-ca" pod "91a89702-510f-436c-8443-b1eba9764b6d" (UID: "91a89702-510f-436c-8443-b1eba9764b6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:31:21.898037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.898016    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:21.899804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.899780    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-xks-ca" pod "91a89702-510f-436c-8443-b1eba9764b6d" (UID: "91a89702-510f-436c-8443-b1eba9764b6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:31:21.899883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.899847    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a89702-510f-436c-8443-b1eba9764b6d-kube-api-access-bjlm9" pod "91a89702-510f-436c-8443-b1eba9764b6d" (UID: "91a89702-510f-436c-8443-b1eba9764b6d"). InnerVolumeSpecName "kube-api-access-bjlm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:31:21.999550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.999466    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjlm9\" (UniqueName: \"kubernetes.io/projected/91a89702-510f-436c-8443-b1eba9764b6d-kube-api-access-bjlm9\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:21.999550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:21.999500    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a89702-510f-436c-8443-b1eba9764b6d-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:22.740262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:22.740235    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"
Nov 03 08:31:22.740404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:22.740236    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf" event={"ID":"91a89702-510f-436c-8443-b1eba9764b6d","Type":"ContainerDied","Data":"27dd63bad915bcbddecdda301a4a8ea370531fa161b92ab44eed1c4b01a54f0d"}
Nov 03 08:31:22.740404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:22.740358    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27dd63bad915bcbddecdda301a4a8ea370531fa161b92ab44eed1c4b01a54f0d"
Nov 03 08:31:28.374266 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:31:28.374229    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:31:28.756452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.756419    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"]
Nov 03 08:31:28.756785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.756773    6430 cpu_manager.go:414] "containerMap: removing container" podUID="91a89702-510f-436c-8443-b1eba9764b6d" containerName="revoke-keys"
Nov 03 08:31:28.756831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.756786    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a89702-510f-436c-8443-b1eba9764b6d" containerName="revoke-keys"
Nov 03 08:31:28.756865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.756854    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a89702-510f-436c-8443-b1eba9764b6d" containerName="revoke-keys"
Nov 03 08:31:28.760127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.760104    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.762856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.762830    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:31:28.763254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.763225    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:31:28.768774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.768739    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"]
Nov 03 08:31:28.858379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.858337    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.858379 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.858375    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qwj\" (UniqueName: \"kubernetes.io/projected/afcd7cc6-4535-4e88-a48b-ea9165199457-kube-api-access-b9qwj\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.858631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.858415    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.959838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.959789    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.960054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.959887    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.960054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.959906    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qwj\" (UniqueName: \"kubernetes.io/projected/afcd7cc6-4535-4e88-a48b-ea9165199457-kube-api-access-b9qwj\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.960506 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.960483    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.962448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.962422    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:28.969792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:28.969767    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qwj\" (UniqueName: \"kubernetes.io/projected/afcd7cc6-4535-4e88-a48b-ea9165199457-kube-api-access-b9qwj\") pod \"maas-api-revoke-sub-d3fba056-d2rgw\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:29.075260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:29.075173    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:29.359120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:29.359023    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"]
Nov 03 08:31:29.362565 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:31:29.362530    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcd7cc6_4535_4e88_a48b_ea9165199457.slice/crio-4537d60e56cab718b5f6624203b2a6aa173dfb2fa5ca9f0b8d34c77c16102c3a WatchSource:0}: Error finding container 4537d60e56cab718b5f6624203b2a6aa173dfb2fa5ca9f0b8d34c77c16102c3a: Status 404 returned error can't find the container with id 4537d60e56cab718b5f6624203b2a6aa173dfb2fa5ca9f0b8d34c77c16102c3a
Nov 03 08:31:29.772622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:29.772587    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw" event={"ID":"afcd7cc6-4535-4e88-a48b-ea9165199457","Type":"ContainerStarted","Data":"5697679ef1c760be0e6a648b36aa8cdb906333c9ec4267465722f1a90d53a483"}
Nov 03 08:31:29.773088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:29.772645    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw" event={"ID":"afcd7cc6-4535-4e88-a48b-ea9165199457","Type":"ContainerStarted","Data":"4537d60e56cab718b5f6624203b2a6aa173dfb2fa5ca9f0b8d34c77c16102c3a"}
Nov 03 08:31:29.789858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:29.789805    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw" podStartSLOduration=1.789789341 podStartE2EDuration="1.789789341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:31:29.788661238 +0000 UTC m=+2239.291496752" watchObservedRunningTime="2026-11-03 08:31:29.789789341 +0000 UTC m=+2239.292624858"
Nov 03 08:31:30.780106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:30.780041    6430 generic.go:356] "Generic (PLEG): container finished" podID="afcd7cc6-4535-4e88-a48b-ea9165199457" containerID="5697679ef1c760be0e6a648b36aa8cdb906333c9ec4267465722f1a90d53a483" exitCode=0
Nov 03 08:31:30.780550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:30.780105    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw" event={"ID":"afcd7cc6-4535-4e88-a48b-ea9165199457","Type":"ContainerDied","Data":"5697679ef1c760be0e6a648b36aa8cdb906333c9ec4267465722f1a90d53a483"}
Nov 03 08:31:32.095643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.095617    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:32.188714 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.188676    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcd7cc6-4535-4e88-a48b-ea9165199457-kube-api-access-b9qwj\" (UniqueName: \"kubernetes.io/projected/afcd7cc6-4535-4e88-a48b-ea9165199457-kube-api-access-b9qwj\") pod \"afcd7cc6-4535-4e88-a48b-ea9165199457\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") "
Nov 03 08:31:32.188889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.188728    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-xks-ca\") pod \"afcd7cc6-4535-4e88-a48b-ea9165199457\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") "
Nov 03 08:31:32.188889 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.188779    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-openshift-ca\") pod \"afcd7cc6-4535-4e88-a48b-ea9165199457\" (UID: \"afcd7cc6-4535-4e88-a48b-ea9165199457\") "
Nov 03 08:31:32.189366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.189320    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-openshift-ca" pod "afcd7cc6-4535-4e88-a48b-ea9165199457" (UID: "afcd7cc6-4535-4e88-a48b-ea9165199457"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:31:32.191005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.190977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcd7cc6-4535-4e88-a48b-ea9165199457-kube-api-access-b9qwj" pod "afcd7cc6-4535-4e88-a48b-ea9165199457" (UID: "afcd7cc6-4535-4e88-a48b-ea9165199457"). InnerVolumeSpecName "kube-api-access-b9qwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:31:32.191216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.191196    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-xks-ca" pod "afcd7cc6-4535-4e88-a48b-ea9165199457" (UID: "afcd7cc6-4535-4e88-a48b-ea9165199457"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:31:32.290381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.290346    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qwj\" (UniqueName: \"kubernetes.io/projected/afcd7cc6-4535-4e88-a48b-ea9165199457-kube-api-access-b9qwj\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:32.290381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.290376    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:32.290381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.290385    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcd7cc6-4535-4e88-a48b-ea9165199457-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:32.789978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.789951    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"
Nov 03 08:31:32.790150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.789948    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw" event={"ID":"afcd7cc6-4535-4e88-a48b-ea9165199457","Type":"ContainerDied","Data":"4537d60e56cab718b5f6624203b2a6aa173dfb2fa5ca9f0b8d34c77c16102c3a"}
Nov 03 08:31:32.790150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:32.790092    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4537d60e56cab718b5f6624203b2a6aa173dfb2fa5ca9f0b8d34c77c16102c3a"
Nov 03 08:31:35.102427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:35.102388    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:31:46.151261 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:46.151229    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:31:49.221675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.221639    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"]
Nov 03 08:31:49.222110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.222000    6430 cpu_manager.go:414] "containerMap: removing container" podUID="afcd7cc6-4535-4e88-a48b-ea9165199457" containerName="revoke-keys"
Nov 03 08:31:49.222110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.222010    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcd7cc6-4535-4e88-a48b-ea9165199457" containerName="revoke-keys"
Nov 03 08:31:49.222110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.222081    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcd7cc6-4535-4e88-a48b-ea9165199457" containerName="revoke-keys"
Nov 03 08:31:49.225150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.225129    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.227844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.227820    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:31:49.227960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.227872    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:31:49.232835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.232815    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"]
Nov 03 08:31:49.349025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.348940    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.349025 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.348980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lphzv\" (UniqueName: \"kubernetes.io/projected/fc857b2e-f99f-4400-852b-7f7b1fb9639b-kube-api-access-lphzv\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.349257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.349148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.450628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.450539    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.450825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.450643    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.450825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.450672    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lphzv\" (UniqueName: \"kubernetes.io/projected/fc857b2e-f99f-4400-852b-7f7b1fb9639b-kube-api-access-lphzv\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.451388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.451365    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.453039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.453020    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.457826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.457808    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lphzv\" (UniqueName: \"kubernetes.io/projected/fc857b2e-f99f-4400-852b-7f7b1fb9639b-kube-api-access-lphzv\") pod \"maas-api-revoke-sub-ae594979-zdsvs\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.541750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.541656    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:49.815037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.814957    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"]
Nov 03 08:31:49.818351 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:31:49.818313    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc857b2e_f99f_4400_852b_7f7b1fb9639b.slice/crio-bb7615b3bea6a1e10e47cbd960d8f038c9b82fd1dd170fdc87447b5ed308ac7e WatchSource:0}: Error finding container bb7615b3bea6a1e10e47cbd960d8f038c9b82fd1dd170fdc87447b5ed308ac7e: Status 404 returned error can't find the container with id bb7615b3bea6a1e10e47cbd960d8f038c9b82fd1dd170fdc87447b5ed308ac7e
Nov 03 08:31:49.889186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:49.889159    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs" event={"ID":"fc857b2e-f99f-4400-852b-7f7b1fb9639b","Type":"ContainerStarted","Data":"bb7615b3bea6a1e10e47cbd960d8f038c9b82fd1dd170fdc87447b5ed308ac7e"}
Nov 03 08:31:50.894965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:50.894868    6430 generic.go:356] "Generic (PLEG): container finished" podID="fc857b2e-f99f-4400-852b-7f7b1fb9639b" containerID="62ff32b41cfd4662f359e94caf7b3ab139825077e68547fd214ee87630d928c0" exitCode=0
Nov 03 08:31:50.895432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:50.894962    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs" event={"ID":"fc857b2e-f99f-4400-852b-7f7b1fb9639b","Type":"ContainerDied","Data":"62ff32b41cfd4662f359e94caf7b3ab139825077e68547fd214ee87630d928c0"}
Nov 03 08:31:52.027367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.027343    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:52.179244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.179162    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-xks-ca\") pod \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") "
Nov 03 08:31:52.179415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.179258    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc857b2e-f99f-4400-852b-7f7b1fb9639b-kube-api-access-lphzv\" (UniqueName: \"kubernetes.io/projected/fc857b2e-f99f-4400-852b-7f7b1fb9639b-kube-api-access-lphzv\") pod \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") "
Nov 03 08:31:52.179415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.179301    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-openshift-ca\") pod \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\" (UID: \"fc857b2e-f99f-4400-852b-7f7b1fb9639b\") "
Nov 03 08:31:52.179718 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.179694    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-openshift-ca" pod "fc857b2e-f99f-4400-852b-7f7b1fb9639b" (UID: "fc857b2e-f99f-4400-852b-7f7b1fb9639b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:31:52.181386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.181363    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-xks-ca" pod "fc857b2e-f99f-4400-852b-7f7b1fb9639b" (UID: "fc857b2e-f99f-4400-852b-7f7b1fb9639b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:31:52.181481 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.181429    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc857b2e-f99f-4400-852b-7f7b1fb9639b-kube-api-access-lphzv" pod "fc857b2e-f99f-4400-852b-7f7b1fb9639b" (UID: "fc857b2e-f99f-4400-852b-7f7b1fb9639b"). InnerVolumeSpecName "kube-api-access-lphzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:31:52.280233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.280206    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:52.280233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.280228    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lphzv\" (UniqueName: \"kubernetes.io/projected/fc857b2e-f99f-4400-852b-7f7b1fb9639b-kube-api-access-lphzv\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:52.280386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.280239    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc857b2e-f99f-4400-852b-7f7b1fb9639b-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:31:52.905632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.905551    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"
Nov 03 08:31:52.905632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.905556    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs" event={"ID":"fc857b2e-f99f-4400-852b-7f7b1fb9639b","Type":"ContainerDied","Data":"bb7615b3bea6a1e10e47cbd960d8f038c9b82fd1dd170fdc87447b5ed308ac7e"}
Nov 03 08:31:52.905632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:31:52.905603    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb7615b3bea6a1e10e47cbd960d8f038c9b82fd1dd170fdc87447b5ed308ac7e"
Nov 03 08:31:58.374771 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:31:58.374724    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:32:05.775113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:05.775079    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:32:07.274237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.274192    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"]
Nov 03 08:32:07.274767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.274747    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fc857b2e-f99f-4400-852b-7f7b1fb9639b" containerName="revoke-keys"
Nov 03 08:32:07.274859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.274771    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc857b2e-f99f-4400-852b-7f7b1fb9639b" containerName="revoke-keys"
Nov 03 08:32:07.274911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.274873    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc857b2e-f99f-4400-852b-7f7b1fb9639b" containerName="revoke-keys"
Nov 03 08:32:07.278151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.278129    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.281618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.281586    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:32:07.281902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.281877    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:32:07.288009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.287973    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"]
Nov 03 08:32:07.420724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.420689    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.420937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.420749    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.420937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.420854    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84t4\" (UniqueName: \"kubernetes.io/projected/86d53509-062d-4e90-8f81-c89bcdef9010-kube-api-access-w84t4\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.521501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.521467    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.521679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.521528    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.521679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.521551    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84t4\" (UniqueName: \"kubernetes.io/projected/86d53509-062d-4e90-8f81-c89bcdef9010-kube-api-access-w84t4\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.522281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.522255    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.524017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.523995    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.528884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.528827    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84t4\" (UniqueName: \"kubernetes.io/projected/86d53509-062d-4e90-8f81-c89bcdef9010-kube-api-access-w84t4\") pod \"maas-api-revoke-sub-a4aef7d7-qkwx6\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.594124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.594094    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:07.868127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.868038    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"]
Nov 03 08:32:07.871236 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:32:07.871209    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d53509_062d_4e90_8f81_c89bcdef9010.slice/crio-6abef3df32c74e480e83d0426d63e80fc20b0b556b09dfe0ed87e07ff48e7a1d WatchSource:0}: Error finding container 6abef3df32c74e480e83d0426d63e80fc20b0b556b09dfe0ed87e07ff48e7a1d: Status 404 returned error can't find the container with id 6abef3df32c74e480e83d0426d63e80fc20b0b556b09dfe0ed87e07ff48e7a1d
Nov 03 08:32:07.989332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:07.989299    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6" event={"ID":"86d53509-062d-4e90-8f81-c89bcdef9010","Type":"ContainerStarted","Data":"6abef3df32c74e480e83d0426d63e80fc20b0b556b09dfe0ed87e07ff48e7a1d"}
Nov 03 08:32:08.995228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:08.995157    6430 generic.go:356] "Generic (PLEG): container finished" podID="86d53509-062d-4e90-8f81-c89bcdef9010" containerID="408ede2b5d38792056fc12fb2d389fe33b90b3f2e10301e1638272b5a674883a" exitCode=0
Nov 03 08:32:08.995595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:08.995242    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6" event={"ID":"86d53509-062d-4e90-8f81-c89bcdef9010","Type":"ContainerDied","Data":"408ede2b5d38792056fc12fb2d389fe33b90b3f2e10301e1638272b5a674883a"}
Nov 03 08:32:10.131381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.131351    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:10.247651 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.247614    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-openshift-ca\") pod \"86d53509-062d-4e90-8f81-c89bcdef9010\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") "
Nov 03 08:32:10.247862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.247669    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d53509-062d-4e90-8f81-c89bcdef9010-kube-api-access-w84t4\" (UniqueName: \"kubernetes.io/projected/86d53509-062d-4e90-8f81-c89bcdef9010-kube-api-access-w84t4\") pod \"86d53509-062d-4e90-8f81-c89bcdef9010\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") "
Nov 03 08:32:10.247862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.247781    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-xks-ca\") pod \"86d53509-062d-4e90-8f81-c89bcdef9010\" (UID: \"86d53509-062d-4e90-8f81-c89bcdef9010\") "
Nov 03 08:32:10.248010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.247977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-openshift-ca" pod "86d53509-062d-4e90-8f81-c89bcdef9010" (UID: "86d53509-062d-4e90-8f81-c89bcdef9010"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:32:10.249979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.249953    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-xks-ca" pod "86d53509-062d-4e90-8f81-c89bcdef9010" (UID: "86d53509-062d-4e90-8f81-c89bcdef9010"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:32:10.250111 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.250000    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d53509-062d-4e90-8f81-c89bcdef9010-kube-api-access-w84t4" pod "86d53509-062d-4e90-8f81-c89bcdef9010" (UID: "86d53509-062d-4e90-8f81-c89bcdef9010"). InnerVolumeSpecName "kube-api-access-w84t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:32:10.348391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.348357    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:10.348391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.348385    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d53509-062d-4e90-8f81-c89bcdef9010-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:10.348391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:10.348395    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84t4\" (UniqueName: \"kubernetes.io/projected/86d53509-062d-4e90-8f81-c89bcdef9010-kube-api-access-w84t4\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:11.005566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:11.005536    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"
Nov 03 08:32:11.005835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:11.005577    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6" event={"ID":"86d53509-062d-4e90-8f81-c89bcdef9010","Type":"ContainerDied","Data":"6abef3df32c74e480e83d0426d63e80fc20b0b556b09dfe0ed87e07ff48e7a1d"}
Nov 03 08:32:11.005835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:11.005654    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6abef3df32c74e480e83d0426d63e80fc20b0b556b09dfe0ed87e07ff48e7a1d"
Nov 03 08:32:14.630893 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:14.630859    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:32:25.846251 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.846216    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"]
Nov 03 08:32:25.846662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.846585    6430 cpu_manager.go:414] "containerMap: removing container" podUID="86d53509-062d-4e90-8f81-c89bcdef9010" containerName="revoke-keys"
Nov 03 08:32:25.846662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.846599    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d53509-062d-4e90-8f81-c89bcdef9010" containerName="revoke-keys"
Nov 03 08:32:25.846744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.846670    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d53509-062d-4e90-8f81-c89bcdef9010" containerName="revoke-keys"
Nov 03 08:32:25.850095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.850074    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:25.856614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.856590    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:32:25.857458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.857431    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:32:25.860477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.860449    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"]
Nov 03 08:32:25.899683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.899645    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7tps\" (UniqueName: \"kubernetes.io/projected/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-kube-api-access-v7tps\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:25.899838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.899690    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:25.899838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:25.899713    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.000878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.000843    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7tps\" (UniqueName: \"kubernetes.io/projected/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-kube-api-access-v7tps\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.001033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.000887    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.001033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.000918    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.001610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.001584    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.003294 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.003275    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.008522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.008503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7tps\" (UniqueName: \"kubernetes.io/projected/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-kube-api-access-v7tps\") pod \"maas-api-revoke-sub-79805ca9-jmw6q\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.174859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.174801    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:26.447313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:26.447281    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"]
Nov 03 08:32:26.451385 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:32:26.451356    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1fa25e_d6d1_4ae5_b37c_95daaf2cdffb.slice/crio-14d4fdfe26bf7d32a150466d3a6a2272d3d94113f99acfaa68fb12b7f886c363 WatchSource:0}: Error finding container 14d4fdfe26bf7d32a150466d3a6a2272d3d94113f99acfaa68fb12b7f886c363: Status 404 returned error can't find the container with id 14d4fdfe26bf7d32a150466d3a6a2272d3d94113f99acfaa68fb12b7f886c363
Nov 03 08:32:27.092315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:27.092283    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q" event={"ID":"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb","Type":"ContainerStarted","Data":"38e6dd0cc4cfd38484adc33c31e4a8b0d93008c334b3e56bd9077fd1a25a4ba5"}
Nov 03 08:32:27.092781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:27.092335    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q" event={"ID":"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb","Type":"ContainerStarted","Data":"14d4fdfe26bf7d32a150466d3a6a2272d3d94113f99acfaa68fb12b7f886c363"}
Nov 03 08:32:27.107916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:27.107865    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q" podStartSLOduration=2.10784865 podStartE2EDuration="2.10784865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:32:27.106680038 +0000 UTC m=+2296.609515556" watchObservedRunningTime="2026-11-03 08:32:27.10784865 +0000 UTC m=+2296.610684165"
Nov 03 08:32:28.097936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:28.097902    6430 generic.go:356] "Generic (PLEG): container finished" podID="0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" containerID="38e6dd0cc4cfd38484adc33c31e4a8b0d93008c334b3e56bd9077fd1a25a4ba5" exitCode=0
Nov 03 08:32:28.098434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:28.097958    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q" event={"ID":"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb","Type":"ContainerDied","Data":"38e6dd0cc4cfd38484adc33c31e4a8b0d93008c334b3e56bd9077fd1a25a4ba5"}
Nov 03 08:32:28.374680 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:32:28.374590    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:32:29.226944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.226920    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:29.325288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.325250    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-openshift-ca\") pod \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") "
Nov 03 08:32:29.325447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.325305    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-xks-ca\") pod \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") "
Nov 03 08:32:29.325447 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.325342    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-kube-api-access-v7tps\" (UniqueName: \"kubernetes.io/projected/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-kube-api-access-v7tps\") pod \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\" (UID: \"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb\") "
Nov 03 08:32:29.325713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.325686    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-openshift-ca" pod "0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" (UID: "0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:32:29.327639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.327610    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-xks-ca" pod "0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" (UID: "0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:32:29.327763 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.327696    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-kube-api-access-v7tps" pod "0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" (UID: "0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb"). InnerVolumeSpecName "kube-api-access-v7tps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:32:29.426086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.425996    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:29.426086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.426025    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:29.426086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:29.426039    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7tps\" (UniqueName: \"kubernetes.io/projected/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb-kube-api-access-v7tps\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:30.108869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:30.108843    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"
Nov 03 08:32:30.109045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:30.108843    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q" event={"ID":"0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb","Type":"ContainerDied","Data":"14d4fdfe26bf7d32a150466d3a6a2272d3d94113f99acfaa68fb12b7f886c363"}
Nov 03 08:32:30.109045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:30.108960    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d4fdfe26bf7d32a150466d3a6a2272d3d94113f99acfaa68fb12b7f886c363"
Nov 03 08:32:43.456936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.456903    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"]
Nov 03 08:32:43.457375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.457267    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" containerName="revoke-keys"
Nov 03 08:32:43.457375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.457277    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" containerName="revoke-keys"
Nov 03 08:32:43.457375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.457361    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" containerName="revoke-keys"
Nov 03 08:32:43.460472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.460451    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.462802 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.462776    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:32:43.463182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.463165    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:32:43.471274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.471252    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"]
Nov 03 08:32:43.594191 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.594148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdldz\" (UniqueName: \"kubernetes.io/projected/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-kube-api-access-wdldz\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.594377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.594218    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.594377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.594290    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.695716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.695675    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.695900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.695745    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdldz\" (UniqueName: \"kubernetes.io/projected/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-kube-api-access-wdldz\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.695900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.695781    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.696444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.696417    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.698415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.698391    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.703146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.703121    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdldz\" (UniqueName: \"kubernetes.io/projected/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-kube-api-access-wdldz\") pod \"maas-api-revoke-sub-0b7e8edb-n86sq\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:43.773153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:43.773023    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:44.048920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:44.048786    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"]
Nov 03 08:32:44.051663 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:32:44.051636    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161a6a4e_79d1_45fe_a53b_cf2cf6f151da.slice/crio-d1be6b696663d2b3d0722a290d6213cd521dd3e73d40d5f35167cc43d532fc0a WatchSource:0}: Error finding container d1be6b696663d2b3d0722a290d6213cd521dd3e73d40d5f35167cc43d532fc0a: Status 404 returned error can't find the container with id d1be6b696663d2b3d0722a290d6213cd521dd3e73d40d5f35167cc43d532fc0a
Nov 03 08:32:44.182753 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:44.182715    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq" event={"ID":"161a6a4e-79d1-45fe-a53b-cf2cf6f151da","Type":"ContainerStarted","Data":"d1be6b696663d2b3d0722a290d6213cd521dd3e73d40d5f35167cc43d532fc0a"}
Nov 03 08:32:45.188546 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:45.188512    6430 generic.go:356] "Generic (PLEG): container finished" podID="161a6a4e-79d1-45fe-a53b-cf2cf6f151da" containerID="be2a3947575f15eb7bbbbd6059a3062b7ce7ec431e6e6577e73fda26dacff6e0" exitCode=0
Nov 03 08:32:45.188925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:45.188565    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq" event={"ID":"161a6a4e-79d1-45fe-a53b-cf2cf6f151da","Type":"ContainerDied","Data":"be2a3947575f15eb7bbbbd6059a3062b7ce7ec431e6e6577e73fda26dacff6e0"}
Nov 03 08:32:46.322253 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.322228    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:46.421370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.421328    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-openshift-ca\") pod \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") "
Nov 03 08:32:46.421526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.421446    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-kube-api-access-wdldz\" (UniqueName: \"kubernetes.io/projected/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-kube-api-access-wdldz\") pod \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") "
Nov 03 08:32:46.421526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.421479    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-xks-ca\") pod \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\" (UID: \"161a6a4e-79d1-45fe-a53b-cf2cf6f151da\") "
Nov 03 08:32:46.421692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.421668    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-openshift-ca" pod "161a6a4e-79d1-45fe-a53b-cf2cf6f151da" (UID: "161a6a4e-79d1-45fe-a53b-cf2cf6f151da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:32:46.421798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.421779    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:46.423659 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.423632    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-xks-ca" pod "161a6a4e-79d1-45fe-a53b-cf2cf6f151da" (UID: "161a6a4e-79d1-45fe-a53b-cf2cf6f151da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:32:46.423745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.423661    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-kube-api-access-wdldz" pod "161a6a4e-79d1-45fe-a53b-cf2cf6f151da" (UID: "161a6a4e-79d1-45fe-a53b-cf2cf6f151da"). InnerVolumeSpecName "kube-api-access-wdldz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:32:46.522205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.522151    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdldz\" (UniqueName: \"kubernetes.io/projected/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-kube-api-access-wdldz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:46.522205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:46.522173    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a6a4e-79d1-45fe-a53b-cf2cf6f151da-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:47.199421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:47.199377    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq" event={"ID":"161a6a4e-79d1-45fe-a53b-cf2cf6f151da","Type":"ContainerDied","Data":"d1be6b696663d2b3d0722a290d6213cd521dd3e73d40d5f35167cc43d532fc0a"}
Nov 03 08:32:47.199421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:47.199412    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"
Nov 03 08:32:47.199648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:47.199476    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1be6b696663d2b3d0722a290d6213cd521dd3e73d40d5f35167cc43d532fc0a"
Nov 03 08:32:54.084860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.084827    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"]
Nov 03 08:32:54.085305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.085221    6430 cpu_manager.go:414] "containerMap: removing container" podUID="161a6a4e-79d1-45fe-a53b-cf2cf6f151da" containerName="revoke-keys"
Nov 03 08:32:54.085305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.085233    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161a6a4e-79d1-45fe-a53b-cf2cf6f151da" containerName="revoke-keys"
Nov 03 08:32:54.085305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.085300    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="161a6a4e-79d1-45fe-a53b-cf2cf6f151da" containerName="revoke-keys"
Nov 03 08:32:54.088353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.088333    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.091634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.091614    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:32:54.091730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.091671    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:32:54.097312 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.097288    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"]
Nov 03 08:32:54.186716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.186677    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.186858 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.186789    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22pnc\" (UniqueName: \"kubernetes.io/projected/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-kube-api-access-22pnc\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.186970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.186952    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.287756 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.287724    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22pnc\" (UniqueName: \"kubernetes.io/projected/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-kube-api-access-22pnc\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.287902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.287859    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.287902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.287891    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.288479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.288461    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.290305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.290287    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.299103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.299050    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22pnc\" (UniqueName: \"kubernetes.io/projected/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-kube-api-access-22pnc\") pod \"maas-api-revoke-sub-3d4156f1-wb5jd\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.401581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.401526    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:54.676161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:54.676078    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"]
Nov 03 08:32:54.679667 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:32:54.679643    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fd9ab7_4815_40af_ab4a_6fbe1f4b2648.slice/crio-003428aa73ba876d779dba1d56ee7be8311ce7730b09899664d27a219417e2b2 WatchSource:0}: Error finding container 003428aa73ba876d779dba1d56ee7be8311ce7730b09899664d27a219417e2b2: Status 404 returned error can't find the container with id 003428aa73ba876d779dba1d56ee7be8311ce7730b09899664d27a219417e2b2
Nov 03 08:32:55.237980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:55.237939    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd" event={"ID":"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648","Type":"ContainerStarted","Data":"8ea6f546523209af61e6124dab2b695e1cbe0e168083e7ec08d3d48ad5c01720"}
Nov 03 08:32:55.238398 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:55.237997    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd" event={"ID":"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648","Type":"ContainerStarted","Data":"003428aa73ba876d779dba1d56ee7be8311ce7730b09899664d27a219417e2b2"}
Nov 03 08:32:55.253323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:55.253263    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd" podStartSLOduration=1.253250688 podStartE2EDuration="1.253250688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:32:55.252561642 +0000 UTC m=+2324.755397157" watchObservedRunningTime="2026-11-03 08:32:55.253250688 +0000 UTC m=+2324.756086202"
Nov 03 08:32:56.244478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:56.244442    6430 generic.go:356] "Generic (PLEG): container finished" podID="b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" containerID="8ea6f546523209af61e6124dab2b695e1cbe0e168083e7ec08d3d48ad5c01720" exitCode=0
Nov 03 08:32:56.244916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:56.244517    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd" event={"ID":"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648","Type":"ContainerDied","Data":"8ea6f546523209af61e6124dab2b695e1cbe0e168083e7ec08d3d48ad5c01720"}
Nov 03 08:32:57.387295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.387266    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:57.516323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.516234    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-xks-ca\") pod \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") "
Nov 03 08:32:57.516323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.516321    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-openshift-ca\") pod \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") "
Nov 03 08:32:57.516525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.516435    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-kube-api-access-22pnc\" (UniqueName: \"kubernetes.io/projected/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-kube-api-access-22pnc\") pod \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\" (UID: \"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648\") "
Nov 03 08:32:57.516736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.516702    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-openshift-ca" pod "b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" (UID: "b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:32:57.518531 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.518509    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-xks-ca" pod "b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" (UID: "b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:32:57.518672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.518654    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-kube-api-access-22pnc" pod "b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" (UID: "b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648"). InnerVolumeSpecName "kube-api-access-22pnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:32:57.617495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.617460    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22pnc\" (UniqueName: \"kubernetes.io/projected/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-kube-api-access-22pnc\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:57.617495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.617489    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:57.617495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:57.617501    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:32:58.258437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:58.258411    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"
Nov 03 08:32:58.258611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:58.258450    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd" event={"ID":"b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648","Type":"ContainerDied","Data":"003428aa73ba876d779dba1d56ee7be8311ce7730b09899664d27a219417e2b2"}
Nov 03 08:32:58.258611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:32:58.258496    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="003428aa73ba876d779dba1d56ee7be8311ce7730b09899664d27a219417e2b2"
Nov 03 08:32:58.374716 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:32:58.374676    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:33:14.690364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:14.690294    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:33:15.069660 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.069572    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"]
Nov 03 08:33:15.075884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.075855    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wb5jd"]
Nov 03 08:33:15.098602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.098568    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"]
Nov 03 08:33:15.098948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.098931    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" containerName="revoke-keys"
Nov 03 08:33:15.098948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.098944    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" containerName="revoke-keys"
Nov 03 08:33:15.099177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.099012    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" containerName="revoke-keys"
Nov 03 08:33:15.102087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.102050    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.111466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.111443    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"]
Nov 03 08:33:15.146855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.146819    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.146855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.146853    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrrl\" (UniqueName: \"kubernetes.io/projected/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-kube-api-access-clrrl\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.147120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.146897    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.151648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.151621    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:33:15.151780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.151703    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:33:15.248161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.248132    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.248161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.248162    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrrl\" (UniqueName: \"kubernetes.io/projected/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-kube-api-access-clrrl\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.248375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.248207    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.248811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.248786    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.250664 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.250641    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.255765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.255740    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrrl\" (UniqueName: \"kubernetes.io/projected/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-kube-api-access-clrrl\") pod \"maas-api-revoke-sub-3d4156f1-2gxx7\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.419141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.419057    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:15.690610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:15.690579    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"]
Nov 03 08:33:15.693920 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:33:15.693895    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15f4d1d_47f1_4bcd_97f9_751ef6aa04fe.slice/crio-d7d7c93640cb2bdb20668478b63a514fe963b63562d8d29630cb94171d0f3965 WatchSource:0}: Error finding container d7d7c93640cb2bdb20668478b63a514fe963b63562d8d29630cb94171d0f3965: Status 404 returned error can't find the container with id d7d7c93640cb2bdb20668478b63a514fe963b63562d8d29630cb94171d0f3965
Nov 03 08:33:16.368644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:16.368556    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7" event={"ID":"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe","Type":"ContainerStarted","Data":"f41b0103e5b84db445e99bfc80b8a6dbccd5692b1b1f472cd5fe039864fcb7c0"}
Nov 03 08:33:16.368644 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:16.368611    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7" event={"ID":"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe","Type":"ContainerStarted","Data":"d7d7c93640cb2bdb20668478b63a514fe963b63562d8d29630cb94171d0f3965"}
Nov 03 08:33:16.384774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:16.384722    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7" podStartSLOduration=1.384708761 podStartE2EDuration="1.384708761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:33:16.382386363 +0000 UTC m=+2345.885221878" watchObservedRunningTime="2026-11-03 08:33:16.384708761 +0000 UTC m=+2345.887544274"
Nov 03 08:33:16.615132 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:16.615101    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648" path="/var/lib/kubelet/pods/b7fd9ab7-4815-40af-ab4a-6fbe1f4b2648/volumes"
Nov 03 08:33:17.374301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:17.374268    6430 generic.go:356] "Generic (PLEG): container finished" podID="b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" containerID="f41b0103e5b84db445e99bfc80b8a6dbccd5692b1b1f472cd5fe039864fcb7c0" exitCode=0
Nov 03 08:33:17.374701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:17.374314    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7" event={"ID":"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe","Type":"ContainerDied","Data":"f41b0103e5b84db445e99bfc80b8a6dbccd5692b1b1f472cd5fe039864fcb7c0"}
Nov 03 08:33:18.508405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.508380    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:18.577007 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.576973    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-xks-ca\") pod \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") "
Nov 03 08:33:18.577174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.577082    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-openshift-ca\") pod \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") "
Nov 03 08:33:18.577174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.577118    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-kube-api-access-clrrl\" (UniqueName: \"kubernetes.io/projected/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-kube-api-access-clrrl\") pod \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\" (UID: \"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe\") "
Nov 03 08:33:18.577440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.577412    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-openshift-ca" pod "b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" (UID: "b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:33:18.579195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.579163    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-xks-ca" pod "b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" (UID: "b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:33:18.579304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.579268    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-kube-api-access-clrrl" pod "b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" (UID: "b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe"). InnerVolumeSpecName "kube-api-access-clrrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:33:18.678666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.678604    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:18.678666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.678627    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrrl\" (UniqueName: \"kubernetes.io/projected/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-kube-api-access-clrrl\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:18.678666 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:18.678637    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:19.385121 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:19.385041    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"
Nov 03 08:33:19.385310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:19.385038    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7" event={"ID":"b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe","Type":"ContainerDied","Data":"d7d7c93640cb2bdb20668478b63a514fe963b63562d8d29630cb94171d0f3965"}
Nov 03 08:33:19.385310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:19.385188    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d7c93640cb2bdb20668478b63a514fe963b63562d8d29630cb94171d0f3965"
Nov 03 08:33:25.219855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:25.219817    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:33:28.374779 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:33:28.374742    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:33:43.020375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:43.020302    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"]
Nov 03 08:33:43.022772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:43.022746    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849668f1-8zxgr"]
Nov 03 08:33:44.615936 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:44.615902    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b220302f-02f5-43e4-96df-e41e25536e8c" path="/var/lib/kubelet/pods/b220302f-02f5-43e4-96df-e41e25536e8c/volumes"
Nov 03 08:33:46.493729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.493694    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"]
Nov 03 08:33:46.494151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.494054    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" containerName="revoke-keys"
Nov 03 08:33:46.494151 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.494081    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" containerName="revoke-keys"
Nov 03 08:33:46.494239 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.494179    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" containerName="revoke-keys"
Nov 03 08:33:46.497422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.497406    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.500144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.500121    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:33:46.500486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.500469    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:33:46.507263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.507239    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"]
Nov 03 08:33:46.641897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.641861    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262j4\" (UniqueName: \"kubernetes.io/projected/d5dba73a-8e41-4962-a36a-d14bd5d6af61-kube-api-access-262j4\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.642088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.641908    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.642088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.642013    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.743155 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.743122    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.743309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.743186    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.743309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.743254    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262j4\" (UniqueName: \"kubernetes.io/projected/d5dba73a-8e41-4962-a36a-d14bd5d6af61-kube-api-access-262j4\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.743806 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.743744    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.745661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.745638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.750688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.750664    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262j4\" (UniqueName: \"kubernetes.io/projected/d5dba73a-8e41-4962-a36a-d14bd5d6af61-kube-api-access-262j4\") pod \"maas-api-revoke-sub-e2fc9c5b-whxsg\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:46.811650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:46.811629    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:47.026216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:47.026144    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"]
Nov 03 08:33:47.028026 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:47.028000    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a57fc5-pcgwr"]
Nov 03 08:33:47.084551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:47.084527    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"]
Nov 03 08:33:47.087966 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:33:47.087940    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dba73a_8e41_4962_a36a_d14bd5d6af61.slice/crio-19a3062e96951a82cc2cf15052a966265838015bf4ec5419ebc462a99720921b WatchSource:0}: Error finding container 19a3062e96951a82cc2cf15052a966265838015bf4ec5419ebc462a99720921b: Status 404 returned error can't find the container with id 19a3062e96951a82cc2cf15052a966265838015bf4ec5419ebc462a99720921b
Nov 03 08:33:47.531897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:47.531867    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg" event={"ID":"d5dba73a-8e41-4962-a36a-d14bd5d6af61","Type":"ContainerStarted","Data":"3d38d263c485a256b31595cdf15dff98ec29fc39bd826ef4b88322e09cdcc344"}
Nov 03 08:33:47.532248 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:47.531917    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg" event={"ID":"d5dba73a-8e41-4962-a36a-d14bd5d6af61","Type":"ContainerStarted","Data":"19a3062e96951a82cc2cf15052a966265838015bf4ec5419ebc462a99720921b"}
Nov 03 08:33:47.547501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:47.547456    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg" podStartSLOduration=1.5474432999999999 podStartE2EDuration="1.5474433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:33:47.545669951 +0000 UTC m=+2377.048505465" watchObservedRunningTime="2026-11-03 08:33:47.5474433 +0000 UTC m=+2377.050278814"
Nov 03 08:33:48.538762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:48.538729    6430 generic.go:356] "Generic (PLEG): container finished" podID="d5dba73a-8e41-4962-a36a-d14bd5d6af61" containerID="3d38d263c485a256b31595cdf15dff98ec29fc39bd826ef4b88322e09cdcc344" exitCode=0
Nov 03 08:33:48.539230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:48.538822    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg" event={"ID":"d5dba73a-8e41-4962-a36a-d14bd5d6af61","Type":"ContainerDied","Data":"3d38d263c485a256b31595cdf15dff98ec29fc39bd826ef4b88322e09cdcc344"}
Nov 03 08:33:48.615867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:48.615832    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aff661b-468f-4669-a378-bde2349dcab3" path="/var/lib/kubelet/pods/5aff661b-468f-4669-a378-bde2349dcab3/volumes"
Nov 03 08:33:49.671566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.671538    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:49.770392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.770364    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-xks-ca\") pod \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") "
Nov 03 08:33:49.770553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.770414    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dba73a-8e41-4962-a36a-d14bd5d6af61-kube-api-access-262j4\" (UniqueName: \"kubernetes.io/projected/d5dba73a-8e41-4962-a36a-d14bd5d6af61-kube-api-access-262j4\") pod \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") "
Nov 03 08:33:49.770553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.770466    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-openshift-ca\") pod \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\" (UID: \"d5dba73a-8e41-4962-a36a-d14bd5d6af61\") "
Nov 03 08:33:49.770942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.770913    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-openshift-ca" pod "d5dba73a-8e41-4962-a36a-d14bd5d6af61" (UID: "d5dba73a-8e41-4962-a36a-d14bd5d6af61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:33:49.772492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.772469    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-xks-ca" pod "d5dba73a-8e41-4962-a36a-d14bd5d6af61" (UID: "d5dba73a-8e41-4962-a36a-d14bd5d6af61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:33:49.772586 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.772559    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dba73a-8e41-4962-a36a-d14bd5d6af61-kube-api-access-262j4" pod "d5dba73a-8e41-4962-a36a-d14bd5d6af61" (UID: "d5dba73a-8e41-4962-a36a-d14bd5d6af61"). InnerVolumeSpecName "kube-api-access-262j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:33:49.872173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.872110    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:49.872173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.872134    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262j4\" (UniqueName: \"kubernetes.io/projected/d5dba73a-8e41-4962-a36a-d14bd5d6af61-kube-api-access-262j4\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:49.872173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:49.872146    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dba73a-8e41-4962-a36a-d14bd5d6af61-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:50.549159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:50.549124    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"
Nov 03 08:33:50.549360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:50.549128    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg" event={"ID":"d5dba73a-8e41-4962-a36a-d14bd5d6af61","Type":"ContainerDied","Data":"19a3062e96951a82cc2cf15052a966265838015bf4ec5419ebc462a99720921b"}
Nov 03 08:33:50.549360 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:50.549255    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19a3062e96951a82cc2cf15052a966265838015bf4ec5419ebc462a99720921b"
Nov 03 08:33:56.689229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.689199    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"]
Nov 03 08:33:56.689612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.689539    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d5dba73a-8e41-4962-a36a-d14bd5d6af61" containerName="revoke-keys"
Nov 03 08:33:56.689612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.689550    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dba73a-8e41-4962-a36a-d14bd5d6af61" containerName="revoke-keys"
Nov 03 08:33:56.689612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.689602    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dba73a-8e41-4962-a36a-d14bd5d6af61" containerName="revoke-keys"
Nov 03 08:33:56.692638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.692622    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.695196 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.695176    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:33:56.695548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.695537    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:33:56.703609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.703570    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"]
Nov 03 08:33:56.830324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.830294    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.830530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.830371    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.830530 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.830466    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtdw\" (UniqueName: \"kubernetes.io/projected/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-kube-api-access-zwtdw\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.931453 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.931421    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.931588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.931482    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.931588 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.931505    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtdw\" (UniqueName: \"kubernetes.io/projected/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-kube-api-access-zwtdw\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.932124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.932104    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.933868 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.933848    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:56.939389 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:56.939348    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtdw\" (UniqueName: \"kubernetes.io/projected/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-kube-api-access-zwtdw\") pod \"maas-api-revoke-sub-5a29efb3-s8vqc\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:57.007270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:57.007237    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:33:57.283050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:57.282919    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"]
Nov 03 08:33:57.285888 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:33:57.285851    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b125a5a_4ee8_44c7_9b7a_6cb836170fad.slice/crio-522b7a91fb6566ccf966937182c7f0c9cb3dac9f8e55fdc1dc045b23e171ef77 WatchSource:0}: Error finding container 522b7a91fb6566ccf966937182c7f0c9cb3dac9f8e55fdc1dc045b23e171ef77: Status 404 returned error can't find the container with id 522b7a91fb6566ccf966937182c7f0c9cb3dac9f8e55fdc1dc045b23e171ef77
Nov 03 08:33:57.581592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:57.581508    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc" event={"ID":"8b125a5a-4ee8-44c7-9b7a-6cb836170fad","Type":"ContainerStarted","Data":"522b7a91fb6566ccf966937182c7f0c9cb3dac9f8e55fdc1dc045b23e171ef77"}
Nov 03 08:33:58.377268 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:33:58.377169    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:33:58.589232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:58.589191    6430 generic.go:356] "Generic (PLEG): container finished" podID="8b125a5a-4ee8-44c7-9b7a-6cb836170fad" containerID="d95cc342f419ecdb8ec781e2aa45d73ab43b7c9b4f15b536ddd14ed9651d9226" exitCode=0
Nov 03 08:33:58.589427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:33:58.589283    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc" event={"ID":"8b125a5a-4ee8-44c7-9b7a-6cb836170fad","Type":"ContainerDied","Data":"d95cc342f419ecdb8ec781e2aa45d73ab43b7c9b4f15b536ddd14ed9651d9226"}
Nov 03 08:34:00.041585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.041259    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:34:00.087370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.087327    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-openshift-ca\") pod \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") "
Nov 03 08:34:00.087610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.087423    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-kube-api-access-zwtdw\" (UniqueName: \"kubernetes.io/projected/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-kube-api-access-zwtdw\") pod \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") "
Nov 03 08:34:00.087610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.087536    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-xks-ca\") pod \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\" (UID: \"8b125a5a-4ee8-44c7-9b7a-6cb836170fad\") "
Nov 03 08:34:00.087760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.087659    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-openshift-ca" pod "8b125a5a-4ee8-44c7-9b7a-6cb836170fad" (UID: "8b125a5a-4ee8-44c7-9b7a-6cb836170fad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:34:00.087827 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.087813    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:34:00.090520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.090488    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-xks-ca" pod "8b125a5a-4ee8-44c7-9b7a-6cb836170fad" (UID: "8b125a5a-4ee8-44c7-9b7a-6cb836170fad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:34:00.090520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.090508    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-kube-api-access-zwtdw" pod "8b125a5a-4ee8-44c7-9b7a-6cb836170fad" (UID: "8b125a5a-4ee8-44c7-9b7a-6cb836170fad"). InnerVolumeSpecName "kube-api-access-zwtdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:34:00.188842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.188795    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtdw\" (UniqueName: \"kubernetes.io/projected/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-kube-api-access-zwtdw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:34:00.188842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.188841    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b125a5a-4ee8-44c7-9b7a-6cb836170fad-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:34:00.602176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.602141    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"
Nov 03 08:34:00.602176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.602167    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc" event={"ID":"8b125a5a-4ee8-44c7-9b7a-6cb836170fad","Type":"ContainerDied","Data":"522b7a91fb6566ccf966937182c7f0c9cb3dac9f8e55fdc1dc045b23e171ef77"}
Nov 03 08:34:00.602386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:00.602254    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="522b7a91fb6566ccf966937182c7f0c9cb3dac9f8e55fdc1dc045b23e171ef77"
Nov 03 08:34:11.964925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.964886    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"]
Nov 03 08:34:11.971046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.971016    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2gxx7"]
Nov 03 08:34:11.985285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.985254    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"]
Nov 03 08:34:11.985696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.985678    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8b125a5a-4ee8-44c7-9b7a-6cb836170fad" containerName="revoke-keys"
Nov 03 08:34:11.985786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.985699    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b125a5a-4ee8-44c7-9b7a-6cb836170fad" containerName="revoke-keys"
Nov 03 08:34:11.985844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.985804    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b125a5a-4ee8-44c7-9b7a-6cb836170fad" containerName="revoke-keys"
Nov 03 08:34:11.988825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.988804    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:11.995019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.994991    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:34:11.995175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.995100    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:34:11.997227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:11.997201    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"]
Nov 03 08:34:12.071980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.071946    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb02345-d31f-4209-ad90-357bc482a113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.072178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.071989    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjx9q\" (UniqueName: \"kubernetes.io/projected/deb02345-d31f-4209-ad90-357bc482a113-kube-api-access-tjx9q\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.072178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.072083    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb02345-d31f-4209-ad90-357bc482a113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.172558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.172506    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb02345-d31f-4209-ad90-357bc482a113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.172558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.172567    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjx9q\" (UniqueName: \"kubernetes.io/projected/deb02345-d31f-4209-ad90-357bc482a113-kube-api-access-tjx9q\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.172769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.172589    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb02345-d31f-4209-ad90-357bc482a113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.173333 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.173306    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb02345-d31f-4209-ad90-357bc482a113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.175128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.175109    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb02345-d31f-4209-ad90-357bc482a113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.180202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.180180    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjx9q\" (UniqueName: \"kubernetes.io/projected/deb02345-d31f-4209-ad90-357bc482a113-kube-api-access-tjx9q\") pod \"maas-api-revoke-sub-3d4156f1-w7zbf\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.303965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.303866    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:12.577323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.577249    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"]
Nov 03 08:34:12.580859 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:34:12.580826    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb02345_d31f_4209_ad90_357bc482a113.slice/crio-621d05e5cf2a062b0be7b79431dc6fc9004b35a48ccd45b7feab0f5839b87f58 WatchSource:0}: Error finding container 621d05e5cf2a062b0be7b79431dc6fc9004b35a48ccd45b7feab0f5839b87f58: Status 404 returned error can't find the container with id 621d05e5cf2a062b0be7b79431dc6fc9004b35a48ccd45b7feab0f5839b87f58
Nov 03 08:34:12.615665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.615635    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe" path="/var/lib/kubelet/pods/b15f4d1d-47f1-4bcd-97f9-751ef6aa04fe/volumes"
Nov 03 08:34:12.655686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:12.655654    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf" event={"ID":"deb02345-d31f-4209-ad90-357bc482a113","Type":"ContainerStarted","Data":"621d05e5cf2a062b0be7b79431dc6fc9004b35a48ccd45b7feab0f5839b87f58"}
Nov 03 08:34:13.662574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:13.662487    6430 generic.go:356] "Generic (PLEG): container finished" podID="deb02345-d31f-4209-ad90-357bc482a113" containerID="06d7e3342b27657d6286db9eb4fbc151cd3fb052530f2349994972e10746595e" exitCode=0
Nov 03 08:34:13.662574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:13.662527    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf" event={"ID":"deb02345-d31f-4209-ad90-357bc482a113","Type":"ContainerDied","Data":"06d7e3342b27657d6286db9eb4fbc151cd3fb052530f2349994972e10746595e"}
Nov 03 08:34:14.740211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.740180    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:14.794645 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.794621    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:14.888751 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.888727    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb02345-d31f-4209-ad90-357bc482a113-kube-api-access-tjx9q\" (UniqueName: \"kubernetes.io/projected/deb02345-d31f-4209-ad90-357bc482a113-kube-api-access-tjx9q\") pod \"deb02345-d31f-4209-ad90-357bc482a113\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") "
Nov 03 08:34:14.888895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.888791    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb02345-d31f-4209-ad90-357bc482a113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb02345-d31f-4209-ad90-357bc482a113-maas-api-openshift-ca\") pod \"deb02345-d31f-4209-ad90-357bc482a113\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") "
Nov 03 08:34:14.888895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.888814    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb02345-d31f-4209-ad90-357bc482a113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb02345-d31f-4209-ad90-357bc482a113-maas-api-xks-ca\") pod \"deb02345-d31f-4209-ad90-357bc482a113\" (UID: \"deb02345-d31f-4209-ad90-357bc482a113\") "
Nov 03 08:34:14.889152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.889131    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb02345-d31f-4209-ad90-357bc482a113-maas-api-openshift-ca" pod "deb02345-d31f-4209-ad90-357bc482a113" (UID: "deb02345-d31f-4209-ad90-357bc482a113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:34:14.890870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.890848    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb02345-d31f-4209-ad90-357bc482a113-kube-api-access-tjx9q" pod "deb02345-d31f-4209-ad90-357bc482a113" (UID: "deb02345-d31f-4209-ad90-357bc482a113"). InnerVolumeSpecName "kube-api-access-tjx9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:34:14.890870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.890847    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb02345-d31f-4209-ad90-357bc482a113-maas-api-xks-ca" pod "deb02345-d31f-4209-ad90-357bc482a113" (UID: "deb02345-d31f-4209-ad90-357bc482a113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:34:14.989397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.989376    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjx9q\" (UniqueName: \"kubernetes.io/projected/deb02345-d31f-4209-ad90-357bc482a113-kube-api-access-tjx9q\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:34:14.989397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.989398    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb02345-d31f-4209-ad90-357bc482a113-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:34:14.989542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:14.989408    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb02345-d31f-4209-ad90-357bc482a113-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:34:15.673090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:15.673045    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"
Nov 03 08:34:15.673249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:15.673092    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf" event={"ID":"deb02345-d31f-4209-ad90-357bc482a113","Type":"ContainerDied","Data":"621d05e5cf2a062b0be7b79431dc6fc9004b35a48ccd45b7feab0f5839b87f58"}
Nov 03 08:34:15.673249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:15.673140    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="621d05e5cf2a062b0be7b79431dc6fc9004b35a48ccd45b7feab0f5839b87f58"
Nov 03 08:34:28.373349 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:34:28.373307    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:34:32.021023 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:32.020989    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"]
Nov 03 08:34:32.024036 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:32.024009    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-94fch"]
Nov 03 08:34:32.616099 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:32.616040    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c7db0c-a0a8-4a62-894c-593bcef7405c" path="/var/lib/kubelet/pods/82c7db0c-a0a8-4a62-894c-593bcef7405c/volumes"
Nov 03 08:34:33.221883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.221844    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:33.570247 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.570150    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"]
Nov 03 08:34:33.570684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.570648    6430 cpu_manager.go:414] "containerMap: removing container" podUID="deb02345-d31f-4209-ad90-357bc482a113" containerName="revoke-keys"
Nov 03 08:34:33.570684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.570667    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb02345-d31f-4209-ad90-357bc482a113" containerName="revoke-keys"
Nov 03 08:34:33.570902 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.570748    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb02345-d31f-4209-ad90-357bc482a113" containerName="revoke-keys"
Nov 03 08:34:33.574245 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.574220    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.580888 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.580865    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0af2f6-openshift-default-dockercfg-6fqpd\""
Nov 03 08:34:33.591552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.591528    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"]
Nov 03 08:34:33.645313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.645228    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:33.652956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.652925    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-podinfo\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.652968    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-data\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.652991    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-socket\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.653076    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-token\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.653146    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-credential-socket\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.653204    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istiod-ca-cert\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.653229    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2d5\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-kube-api-access-mc2d5\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.653338    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-certs\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.653408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.653389    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-envoy\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754285    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istiod-ca-cert\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754344    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2d5\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-kube-api-access-mc2d5\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754383    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-certs\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754414    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-envoy\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754468    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-podinfo\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754502    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-data\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754531    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-socket\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754565    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-token\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754622    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-credential-socket\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.754923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754851    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-certs\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.755205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.754965    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-credential-socket\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.755580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.755500    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istiod-ca-cert\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.755580 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.755545    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-socket\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.755782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.755761    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-data\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.758855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.758790    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-envoy\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.759400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.759367    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-podinfo\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.763474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.763417    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-token\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.763657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.763620    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2d5\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-kube-api-access-mc2d5\") pod \"e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") " pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:33.766303 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.766273    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:33.890679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:33.890594    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:34.176265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.176238    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"]
Nov 03 08:34:34.178518 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:34:34.178488    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7c573a_b82d_4bb3_bc71_ada5109471e3.slice/crio-e310d24641c61120714c841dd9fc1d5a2f527d1915e22b00963af533e9d5da03 WatchSource:0}: Error finding container e310d24641c61120714c841dd9fc1d5a2f527d1915e22b00963af533e9d5da03: Status 404 returned error can't find the container with id e310d24641c61120714c841dd9fc1d5a2f527d1915e22b00963af533e9d5da03
Nov 03 08:34:34.180782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.180736    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:34:34.180877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.180838    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:34:34.180926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.180888    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:34:34.787582 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.787553    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" event={"ID":"6d7c573a-b82d-4bb3-bc71-ada5109471e3","Type":"ContainerStarted","Data":"47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc"}
Nov 03 08:34:34.788108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.787605    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" event={"ID":"6d7c573a-b82d-4bb3-bc71-ada5109471e3","Type":"ContainerStarted","Data":"e310d24641c61120714c841dd9fc1d5a2f527d1915e22b00963af533e9d5da03"}
Nov 03 08:34:34.805851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.805805    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" podStartSLOduration=1.805791949 podStartE2EDuration="1.805791949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:34:34.803727458 +0000 UTC m=+2424.306562972" watchObservedRunningTime="2026-11-03 08:34:34.805791949 +0000 UTC m=+2424.308627462"
Nov 03 08:34:34.891085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:34.891047    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:35.896445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:35.896418    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:36.797786 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:36.797748    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:36.798774 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:36.798758    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:37.024238 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:37.024207    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:37.156368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:37.156265    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:38.167988 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.167946    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"]
Nov 03 08:34:38.171320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.171297    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.177845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.177811    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0af2f6-serving-cert\""
Nov 03 08:34:38.178310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.178287    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7nz66\""
Nov 03 08:34:38.178438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.178409    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 08:34:38.184096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.184030    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"]
Nov 03 08:34:38.252009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.251971    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:38.282364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.282323    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"]
Nov 03 08:34:38.285957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.285936    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.289080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.289036    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0af2f6\""
Nov 03 08:34:38.289227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.289111    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1b4a65ee-dockercfg-v5zjs\""
Nov 03 08:34:38.295915 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.295884    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.296051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.295937    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.296051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.295975    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljfx\" (UniqueName: \"kubernetes.io/projected/e9f46534-edb9-4ee5-87cf-10853efcb7c2-kube-api-access-sljfx\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.296051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.296008    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f46534-edb9-4ee5-87cf-10853efcb7c2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.296584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.296564    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"]
Nov 03 08:34:38.326244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.326201    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"]
Nov 03 08:34:38.329792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.329775    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.339331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.339304    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"]
Nov 03 08:34:38.385844 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.385805    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:38.397171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfct\" (UniqueName: \"kubernetes.io/projected/9c1c5a17-29f1-4d82-8d2a-6750631d191a-kube-api-access-zlfct\") pod \"payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.397339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397251    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.397339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397292    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.397339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397326    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e33572d-a87b-4688-adf8-ed7f1b1944f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.397478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397347    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljfx\" (UniqueName: \"kubernetes.io/projected/e9f46534-edb9-4ee5-87cf-10853efcb7c2-kube-api-access-sljfx\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.397478 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397421    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f46534-edb9-4ee5-87cf-10853efcb7c2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.397551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397482    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99kg9\" (UniqueName: \"kubernetes.io/projected/8e33572d-a87b-4688-adf8-ed7f1b1944f3-kube-api-access-99kg9\") pod \"payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.397551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.397504    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1c5a17-29f1-4d82-8d2a-6750631d191a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.398262 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.398232    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f46534-edb9-4ee5-87cf-10853efcb7c2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.399969 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.399943    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.400129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.400097    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.405843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.405823    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljfx\" (UniqueName: \"kubernetes.io/projected/e9f46534-edb9-4ee5-87cf-10853efcb7c2-kube-api-access-sljfx\") pod \"maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.487768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.487728    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:38.498417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.498378    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfct\" (UniqueName: \"kubernetes.io/projected/9c1c5a17-29f1-4d82-8d2a-6750631d191a-kube-api-access-zlfct\") pod \"payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.498579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.498469    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e33572d-a87b-4688-adf8-ed7f1b1944f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.498758 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.498723    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kg9\" (UniqueName: \"kubernetes.io/projected/8e33572d-a87b-4688-adf8-ed7f1b1944f3-kube-api-access-99kg9\") pod \"payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.498832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.498782    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1c5a17-29f1-4d82-8d2a-6750631d191a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.499285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.499261    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:38.500711 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.500685    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e33572d-a87b-4688-adf8-ed7f1b1944f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.504093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.501446    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1c5a17-29f1-4d82-8d2a-6750631d191a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.510319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.510223    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99kg9\" (UniqueName: \"kubernetes.io/projected/8e33572d-a87b-4688-adf8-ed7f1b1944f3-kube-api-access-99kg9\") pod \"payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.512446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.512420    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfct\" (UniqueName: \"kubernetes.io/projected/9c1c5a17-29f1-4d82-8d2a-6750631d191a-kube-api-access-zlfct\") pod \"payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.599579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.599544    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:38.643920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.643885    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:38.778124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.778031    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"]
Nov 03 08:34:38.782036 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:34:38.782006    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f46534_edb9_4ee5_87cf_10853efcb7c2.slice/crio-ab1a2a591db3d60570b545b2cac4beb8300d443c216ef6d17959ce3e09ff7ef6 WatchSource:0}: Error finding container ab1a2a591db3d60570b545b2cac4beb8300d443c216ef6d17959ce3e09ff7ef6: Status 404 returned error can't find the container with id ab1a2a591db3d60570b545b2cac4beb8300d443c216ef6d17959ce3e09ff7ef6
Nov 03 08:34:38.810594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.810560    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm" event={"ID":"e9f46534-edb9-4ee5-87cf-10853efcb7c2","Type":"ContainerStarted","Data":"ab1a2a591db3d60570b545b2cac4beb8300d443c216ef6d17959ce3e09ff7ef6"}
Nov 03 08:34:38.882867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.882835    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"]
Nov 03 08:34:38.885800 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:34:38.885776    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e33572d_a87b_4688_adf8_ed7f1b1944f3.slice/crio-011c70a0d9597ada568a60cd9b2d4a41e643ea6c851ce558e00d581a0192cb21 WatchSource:0}: Error finding container 011c70a0d9597ada568a60cd9b2d4a41e643ea6c851ce558e00d581a0192cb21: Status 404 returned error can't find the container with id 011c70a0d9597ada568a60cd9b2d4a41e643ea6c851ce558e00d581a0192cb21
Nov 03 08:34:38.949891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:38.949868    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"]
Nov 03 08:34:38.952236 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:34:38.952210    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1c5a17_29f1_4d82_8d2a_6750631d191a.slice/crio-1586c8bd087665b509e359d9c995d233ca4ccca552d7e5498c9b42f11efd24b6 WatchSource:0}: Error finding container 1586c8bd087665b509e359d9c995d233ca4ccca552d7e5498c9b42f11efd24b6: Status 404 returned error can't find the container with id 1586c8bd087665b509e359d9c995d233ca4ccca552d7e5498c9b42f11efd24b6
Nov 03 08:34:39.053237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.053135    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:39.643449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.643349    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:39.817044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.817004    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" event={"ID":"9c1c5a17-29f1-4d82-8d2a-6750631d191a","Type":"ContainerStarted","Data":"70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3"}
Nov 03 08:34:39.817224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.817093    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:39.817224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.817117    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" event={"ID":"9c1c5a17-29f1-4d82-8d2a-6750631d191a","Type":"ContainerStarted","Data":"1586c8bd087665b509e359d9c995d233ca4ccca552d7e5498c9b42f11efd24b6"}
Nov 03 08:34:39.818626 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.818600    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm" event={"ID":"e9f46534-edb9-4ee5-87cf-10853efcb7c2","Type":"ContainerStarted","Data":"d6299ea1ddaaa91f160d6b1c15bec0db84ae5dbb0f9e5d1d14c27143b6ee89a1"}
Nov 03 08:34:39.818766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.818645    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:39.820365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.820338    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" event={"ID":"8e33572d-a87b-4688-adf8-ed7f1b1944f3","Type":"ContainerStarted","Data":"b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220"}
Nov 03 08:34:39.820469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.820388    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" event={"ID":"8e33572d-a87b-4688-adf8-ed7f1b1944f3","Type":"ContainerStarted","Data":"011c70a0d9597ada568a60cd9b2d4a41e643ea6c851ce558e00d581a0192cb21"}
Nov 03 08:34:39.820469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.820418    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:39.832724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.832634    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" podStartSLOduration=1.8326197739999999 podStartE2EDuration="1.832619774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:34:39.832227302 +0000 UTC m=+2429.335062816" watchObservedRunningTime="2026-11-03 08:34:39.832619774 +0000 UTC m=+2429.335455287"
Nov 03 08:34:39.848229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.848128    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" podStartSLOduration=1.8481139739999999 podStartE2EDuration="1.848113974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:34:39.846127856 +0000 UTC m=+2429.348963367" watchObservedRunningTime="2026-11-03 08:34:39.848113974 +0000 UTC m=+2429.350949486"
Nov 03 08:34:39.863887 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:39.863854    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm" podStartSLOduration=1.276228332 podStartE2EDuration="1.863844867s" totalImagesPullingTime="587.616535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:34:39.863103491 +0000 UTC m=+2429.365939005" watchObservedRunningTime="2026-11-03 08:34:39.863844867 +0000 UTC m=+2429.366680384"
Nov 03 08:34:40.252350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:40.252320    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:41.054355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:41.054318    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:41.881452 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:41.881422    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:42.530738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:42.530708    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:43.169966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:43.169924    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:43.923193 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:43.923151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:44.484047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:44.483999    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:45.095725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:45.095690    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:45.659973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:45.659935    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:45.833411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:45.833385    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:34:46.334920 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:46.334890    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:47.128412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:47.128334    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:47.728297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:47.728259    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:48.354669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:48.354631    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:48.943741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:48.943698    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:49.540716 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:49.540674    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:50.145609 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.145535    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:50.735305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.735265    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:50.831358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.831332    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:34:50.831537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.831402    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:34:50.885165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.885134    6430 generic.go:356] "Generic (PLEG): container finished" podID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerID="47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc" exitCode=0
Nov 03 08:34:50.885381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.885197    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" event={"ID":"6d7c573a-b82d-4bb3-bc71-ada5109471e3","Type":"ContainerDied","Data":"47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc"}
Nov 03 08:34:50.885627 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.885609    6430 scope.go:122] "RemoveContainer" containerID="47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc"
Nov 03 08:34:50.886970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.886941    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:34:50.887090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.887006    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:34:50.887090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:50.887040    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:34:51.345122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:51.345088    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:51.812088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:51.807175    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:51.891837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:51.891799    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" event={"ID":"6d7c573a-b82d-4bb3-bc71-ada5109471e3","Type":"ContainerStarted","Data":"56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7"}
Nov 03 08:34:51.967473 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:51.967440    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:52.091938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:52.091852    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:52.579526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:52.579491    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:52.732276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:52.729605    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:52.891088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:52.890986    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:52.892313 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:52.892278    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:34:52.892468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:52.892335    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 08:34:52.943174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:52.943144    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:53.514087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:53.513522    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:53.891541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:53.891457    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:34:53.891541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:53.891515    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 08:34:54.171165 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.171076    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:54.642977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.642935    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"]
Nov 03 08:34:54.646631 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.646603    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.657109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.657086    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0af2f6\"/\"openshift-service-ca.crt\""
Nov 03 08:34:54.657878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.657836    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0af2f6\"/\"kube-root-ca.crt\""
Nov 03 08:34:54.660244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.660205    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0af2f6\"/\"default-dockercfg-qtkhz\""
Nov 03 08:34:54.660518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.660493    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0af2f6\"/\"simulator-single-main-0af2f6-kserve-self-signed-certs\""
Nov 03 08:34:54.661391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.661368    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"]
Nov 03 08:34:54.724585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.724552    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:54.761674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.761632    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tmp-dir\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.761869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.761695    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-dshm\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.761869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.761783    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-model-cache\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.761869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.761826    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tls-certs\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.762083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.761923    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-home\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.762083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.761971    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dks\" (UniqueName: \"kubernetes.io/projected/5b8ac031-b54e-4a47-b0ad-01f9faec501e-kube-api-access-44dks\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.862639    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-dshm\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.862705    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-model-cache\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.863102    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-model-cache\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.863162    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tls-certs\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.863201    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-home\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.863231    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dks\" (UniqueName: \"kubernetes.io/projected/5b8ac031-b54e-4a47-b0ad-01f9faec501e-kube-api-access-44dks\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863538 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.863289    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tmp-dir\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.863593    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tmp-dir\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.863863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.863835    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-home\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.865127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.865106    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-dshm\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.865734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.865714    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tls-certs\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.872700 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.872676    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dks\" (UniqueName: \"kubernetes.io/projected/5b8ac031-b54e-4a47-b0ad-01f9faec501e-kube-api-access-44dks\") pod \"simulator-single-main-0af2f6-kserve-75b8db47df-l2lls\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") " pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:54.891710 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.891684    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:34:54.892143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.891746    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 08:34:54.955184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.955148    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:54.963096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:54.963024    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:55.270680 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.270652    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"]
Nov 03 08:34:55.653939 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.653858    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:55.891491 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.891455    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:34:55.891663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.891515    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 08:34:55.932444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.932411    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls" event={"ID":"5b8ac031-b54e-4a47-b0ad-01f9faec501e","Type":"ContainerStarted","Data":"f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f"}
Nov 03 08:34:55.932906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.932453    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls" event={"ID":"5b8ac031-b54e-4a47-b0ad-01f9faec501e","Type":"ContainerStarted","Data":"15b2fea3cad2443a323cce9c7b72258867761ca1dfafd39b0a6a9b19dc4de240"}
Nov 03 08:34:55.947454 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.947397    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls" podStartSLOduration=1.947379749 podStartE2EDuration="1.947379749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:34:55.947241287 +0000 UTC m=+2445.450076817" watchObservedRunningTime="2026-11-03 08:34:55.947379749 +0000 UTC m=+2445.450215263"
Nov 03 08:34:55.963809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:55.963779    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:34:56.359847 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:56.359759    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:57.068301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:57.068262    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:57.360143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:57.360016    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:57.360809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:57.360786    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:57.363388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:57.363367    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:34:57.750653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:57.750587    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:58.377198 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:34:58.377158    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:34:58.401000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:58.400952    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:58.961850 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:58.961818    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:34:59.736912 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:34:59.736873    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:00.347202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:00.347159    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:00.945953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:00.945919    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:01.552856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:01.552819    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:02.147707 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:02.147666    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:02.737056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:02.737022    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:03.437636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:03.437605    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:04.145883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:04.145851    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:04.732859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:04.732798    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:04.963236 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:04.963208    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:35:04.975534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:04.975511    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:35:04.991552 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:04.991486    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:35:05.345919 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:05.345830    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:05.939406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:05.939353    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:06.540657 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:06.540616    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:07.134358 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:07.134320    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:07.739686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:07.739649    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:08.345934 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:08.345892    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:08.953214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:08.953166    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:09.696968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:09.696931    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:10.380586 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:10.380472    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:11.045807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:11.045775    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:11.778420 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:11.778380    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:12.452175 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:12.452136    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:12.677450 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:12.677414    6430 scope.go:122] "RemoveContainer" containerID="575967de530b420c16da7dec75bb11a9764102066ff49ebbe404c6b71ae65e6f"
Nov 03 08:35:12.686977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:12.686956    6430 scope.go:122] "RemoveContainer" containerID="f751bbfa2c981fe2c584b9433637d0299850111dfd8792318d70beccc7863d68"
Nov 03 08:35:13.206564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:13.206530    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:13.862407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:13.862370    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:14.441674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:14.441639    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:14.792635 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:14.792557    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:15.056184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:15.056105    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:15.649824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:15.649787    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:16.244143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:16.244106    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:16.845867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:16.845819    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:17.445436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:17.445402    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:18.045471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:18.045437    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:18.656051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:18.656021    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:19.330803 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:19.330762    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:19.935413 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:19.935366    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:20.626286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:20.626249    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:21.243581 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:21.243546    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:21.739946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:21.739915    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:22.343798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:22.343764    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:22.945372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:22.945333    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:23.652414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:23.652380    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:24.292759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:24.292717    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:24.857390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:24.857356    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:25.457983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:25.457938    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:26.026601 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:26.026562    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"]
Nov 03 08:35:26.029300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:26.029275    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nn9gr"]
Nov 03 08:35:26.159013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:26.158967    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:26.616323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:26.616292    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57143e0-6ac9-4fde-855e-8bb7e43fd1e2" path="/var/lib/kubelet/pods/e57143e0-6ac9-4fde-855e-8bb7e43fd1e2/volumes"
Nov 03 08:35:26.734793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:26.734758    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:27.384525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:27.384489    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:27.951800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:27.951770    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:28.374947 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:35:28.374864    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:35:28.540585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:28.540548    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:29.164001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:29.163961    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:29.747372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:29.747334    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:30.330319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:30.330274    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:30.936357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:30.936324    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:31.563879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:31.563801    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:32.246956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:32.246922    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:32.934565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:32.934530    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:33.508790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:33.508758    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:34.156011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:34.155967    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:34.741072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:34.741026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:35.400965 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:35.400932    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:36.031768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:36.031736    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:36.633829 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:36.633800    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:36.789345 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:36.789308    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:37.417474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:37.417435    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:38.035752 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:38.035722    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:38.661513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:38.661482    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:39.242978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:39.242946    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:39.836241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:39.836212    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:40.437050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:40.437013    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:41.031782 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:41.031737    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:41.631997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:41.631962    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:41.735739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:41.735707    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"]
Nov 03 08:35:41.831735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:41.831700    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:42.237903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:42.237849    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:42.832579 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:42.832546    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:43.446295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:43.446262    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:44.225608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:44.225537    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:44.828837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:44.828803    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:45.026107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:45.026052    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"]
Nov 03 08:35:45.028598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:45.028566    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5ml2s"]
Nov 03 08:35:45.446974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:45.446940    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:46.039319 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:46.039288    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:46.615438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:46.615406    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9084f7d-a9fe-4619-a561-71b4f5cd5b5f" path="/var/lib/kubelet/pods/f9084f7d-a9fe-4619-a561-71b4f5cd5b5f/volumes"
Nov 03 08:35:46.627206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:46.627152    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:47.039994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.039952    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"]
Nov 03 08:35:47.040903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.040846    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm" podUID="e9f46534-edb9-4ee5-87cf-10853efcb7c2" containerName="maas-api" containerID="cri-o://d6299ea1ddaaa91f160d6b1c15bec0db84ae5dbb0f9e5d1d14c27143b6ee89a1" gracePeriod=30
Nov 03 08:35:47.048811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.048742    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"]
Nov 03 08:35:47.052282 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.052253    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"]
Nov 03 08:35:47.118584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.118537    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:47.203632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.203589    6430 generic.go:356] "Generic (PLEG): container finished" podID="e9f46534-edb9-4ee5-87cf-10853efcb7c2" containerID="d6299ea1ddaaa91f160d6b1c15bec0db84ae5dbb0f9e5d1d14c27143b6ee89a1" exitCode=0
Nov 03 08:35:47.203821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.203638    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm" event={"ID":"e9f46534-edb9-4ee5-87cf-10853efcb7c2","Type":"ContainerDied","Data":"d6299ea1ddaaa91f160d6b1c15bec0db84ae5dbb0f9e5d1d14c27143b6ee89a1"}
Nov 03 08:35:47.233921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.233893    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:47.310958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.310935    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:35:47.443970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.443934    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-maas-api-tls\") pod \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") "
Nov 03 08:35:47.444198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.444033    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-metrics-certs\") pod \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") "
Nov 03 08:35:47.444198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.444108    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f46534-edb9-4ee5-87cf-10853efcb7c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f46534-edb9-4ee5-87cf-10853efcb7c2-openshift-service-ca\") pod \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") "
Nov 03 08:35:47.444198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.444140    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f46534-edb9-4ee5-87cf-10853efcb7c2-kube-api-access-sljfx\" (UniqueName: \"kubernetes.io/projected/e9f46534-edb9-4ee5-87cf-10853efcb7c2-kube-api-access-sljfx\") pod \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\" (UID: \"e9f46534-edb9-4ee5-87cf-10853efcb7c2\") "
Nov 03 08:35:47.444537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.444506    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f46534-edb9-4ee5-87cf-10853efcb7c2-openshift-service-ca" pod "e9f46534-edb9-4ee5-87cf-10853efcb7c2" (UID: "e9f46534-edb9-4ee5-87cf-10853efcb7c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:35:47.446641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.446422    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-maas-api-tls" pod "e9f46534-edb9-4ee5-87cf-10853efcb7c2" (UID: "e9f46534-edb9-4ee5-87cf-10853efcb7c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:35:47.446641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.446528    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f46534-edb9-4ee5-87cf-10853efcb7c2-kube-api-access-sljfx" pod "e9f46534-edb9-4ee5-87cf-10853efcb7c2" (UID: "e9f46534-edb9-4ee5-87cf-10853efcb7c2"). InnerVolumeSpecName "kube-api-access-sljfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:35:47.446809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.446662    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-metrics-certs" pod "e9f46534-edb9-4ee5-87cf-10853efcb7c2" (UID: "e9f46534-edb9-4ee5-87cf-10853efcb7c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:35:47.545494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.545432    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:47.545494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.545461    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f46534-edb9-4ee5-87cf-10853efcb7c2-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:47.545494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.545472    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljfx\" (UniqueName: \"kubernetes.io/projected/e9f46534-edb9-4ee5-87cf-10853efcb7c2-kube-api-access-sljfx\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:47.545494 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.545482    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f46534-edb9-4ee5-87cf-10853efcb7c2-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:47.564174 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.564143    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:47.680957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:47.680923    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:48.048237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:48.048206    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:48.209691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:48.209663    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"
Nov 03 08:35:48.209863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:48.209659    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm" event={"ID":"e9f46534-edb9-4ee5-87cf-10853efcb7c2","Type":"ContainerDied","Data":"ab1a2a591db3d60570b545b2cac4beb8300d443c216ef6d17959ce3e09ff7ef6"}
Nov 03 08:35:48.209863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:48.209794    6430 scope.go:122] "RemoveContainer" containerID="d6299ea1ddaaa91f160d6b1c15bec0db84ae5dbb0f9e5d1d14c27143b6ee89a1"
Nov 03 08:35:48.233051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:48.233029    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"]
Nov 03 08:35:48.237522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:48.237498    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0af2f6-7bcbf8f6f-tq7dm"]
Nov 03 08:35:48.615325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:48.615290    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f46534-edb9-4ee5-87cf-10853efcb7c2" path="/var/lib/kubelet/pods/e9f46534-edb9-4ee5-87cf-10853efcb7c2/volumes"
Nov 03 08:35:52.049814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:52.049769    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" podUID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" containerName="payload-processing" containerID="cri-o://70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3" gracePeriod=25
Nov 03 08:35:52.053355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:52.053323    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" podUID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" containerName="payload-pre-processing" containerID="cri-o://b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220" gracePeriod=25
Nov 03 08:35:53.687691 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:53.687638    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:53.807480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:53.807442    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:53.950256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:53.950159    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"]
Nov 03 08:35:53.950500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:53.950462    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy" containerID="cri-o://56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7" gracePeriod=30
Nov 03 08:35:54.012776 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:54.012742    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:35:55.108001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:55.107968    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"]
Nov 03 08:35:55.112979 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:55.112953    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w7zbf"]
Nov 03 08:35:56.021632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:56.021604    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"]
Nov 03 08:35:56.023551 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:56.023529    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w9jkj"]
Nov 03 08:35:56.615736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:56.615702    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c29096-7b1f-4132-b9b0-12eca3efa44f" path="/var/lib/kubelet/pods/c1c29096-7b1f-4132-b9b0-12eca3efa44f/volumes"
Nov 03 08:35:56.616160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:56.616128    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb02345-d31f-4209-ad90-357bc482a113" path="/var/lib/kubelet/pods/deb02345-d31f-4209-ad90-357bc482a113/volumes"
Nov 03 08:35:56.756216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:56.756158    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls" podUID="5b8ac031-b54e-4a47-b0ad-01f9faec501e" containerName="main" containerID="cri-o://f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f" gracePeriod=45
Nov 03 08:35:57.005814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.005783    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:35:57.137519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137420    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tmp-dir\") pod \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") "
Nov 03 08:35:57.137679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137523    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-model-cache\") pod \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") "
Nov 03 08:35:57.137679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137620    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-dshm\") pod \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") "
Nov 03 08:35:57.137679 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137669    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tls-certs\") pod \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") "
Nov 03 08:35:57.137870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137724    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-home\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-home\") pod \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") "
Nov 03 08:35:57.137870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137751    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tmp-dir" pod "5b8ac031-b54e-4a47-b0ad-01f9faec501e" (UID: "5b8ac031-b54e-4a47-b0ad-01f9faec501e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:57.137870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137767    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8ac031-b54e-4a47-b0ad-01f9faec501e-kube-api-access-44dks\" (UniqueName: \"kubernetes.io/projected/5b8ac031-b54e-4a47-b0ad-01f9faec501e-kube-api-access-44dks\") pod \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\" (UID: \"5b8ac031-b54e-4a47-b0ad-01f9faec501e\") "
Nov 03 08:35:57.137870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.137760    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-model-cache" pod "5b8ac031-b54e-4a47-b0ad-01f9faec501e" (UID: "5b8ac031-b54e-4a47-b0ad-01f9faec501e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:57.138098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.138002    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-home" pod "5b8ac031-b54e-4a47-b0ad-01f9faec501e" (UID: "5b8ac031-b54e-4a47-b0ad-01f9faec501e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:57.138180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.138152    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:57.138180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.138167    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:57.138180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.138176    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:57.140004 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.139974    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-dshm" pod "5b8ac031-b54e-4a47-b0ad-01f9faec501e" (UID: "5b8ac031-b54e-4a47-b0ad-01f9faec501e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:57.140156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.140032    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tls-certs" pod "5b8ac031-b54e-4a47-b0ad-01f9faec501e" (UID: "5b8ac031-b54e-4a47-b0ad-01f9faec501e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:35:57.140548 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.140533    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8ac031-b54e-4a47-b0ad-01f9faec501e-kube-api-access-44dks" pod "5b8ac031-b54e-4a47-b0ad-01f9faec501e" (UID: "5b8ac031-b54e-4a47-b0ad-01f9faec501e"). InnerVolumeSpecName "kube-api-access-44dks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:35:57.239001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.238958    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44dks\" (UniqueName: \"kubernetes.io/projected/5b8ac031-b54e-4a47-b0ad-01f9faec501e-kube-api-access-44dks\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:57.239001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.238993    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8ac031-b54e-4a47-b0ad-01f9faec501e-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:57.239001 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.239004    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8ac031-b54e-4a47-b0ad-01f9faec501e-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:57.255978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.255948    6430 generic.go:356] "Generic (PLEG): container finished" podID="5b8ac031-b54e-4a47-b0ad-01f9faec501e" containerID="f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f" exitCode=0
Nov 03 08:35:57.256184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.255992    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"
Nov 03 08:35:57.256184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.256024    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls" event={"ID":"5b8ac031-b54e-4a47-b0ad-01f9faec501e","Type":"ContainerDied","Data":"f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f"}
Nov 03 08:35:57.256184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.256082    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls" event={"ID":"5b8ac031-b54e-4a47-b0ad-01f9faec501e","Type":"ContainerDied","Data":"15b2fea3cad2443a323cce9c7b72258867761ca1dfafd39b0a6a9b19dc4de240"}
Nov 03 08:35:57.256184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.256107    6430 scope.go:122] "RemoveContainer" containerID="f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f"
Nov 03 08:35:57.265499 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.265477    6430 scope.go:122] "RemoveContainer" containerID="f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f"
Nov 03 08:35:57.265753 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:35:57.265735    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f\": container with ID starting with f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f not found: ID does not exist" containerID="f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f"
Nov 03 08:35:57.265804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.265762    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f"} err="failed to get container status \"f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f\": rpc error: code = NotFound desc = could not find container \"f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f\": container with ID starting with f3cdd785011e794aa0d9ec46cb5f510f281afe183a916668f4c7be77a6c1017f not found: ID does not exist"
Nov 03 08:35:57.275795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.275767    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"]
Nov 03 08:35:57.278648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.278626    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls"]
Nov 03 08:35:57.362872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.362828    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:35:57.363039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.362885    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:35:57.981813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:57.981768    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-0af2f6/simulator-single-main-0af2f6-kserve-75b8db47df-l2lls" podUID="5b8ac031-b54e-4a47-b0ad-01f9faec501e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.196:8000/ready\": context deadline exceeded"
Nov 03 08:35:58.374524 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:35:58.374442    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:35:58.615670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:58.615626    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8ac031-b54e-4a47-b0ad-01f9faec501e" path="/var/lib/kubelet/pods/5b8ac031-b54e-4a47-b0ad-01f9faec501e/volumes"
Nov 03 08:35:59.196829 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.196802    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:35:59.268634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.268546    6430 generic.go:356] "Generic (PLEG): container finished" podID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerID="56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7" exitCode=0
Nov 03 08:35:59.268634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.268600    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"
Nov 03 08:35:59.268634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.268603    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" event={"ID":"6d7c573a-b82d-4bb3-bc71-ada5109471e3","Type":"ContainerDied","Data":"56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7"}
Nov 03 08:35:59.268866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.268655    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt" event={"ID":"6d7c573a-b82d-4bb3-bc71-ada5109471e3","Type":"ContainerDied","Data":"e310d24641c61120714c841dd9fc1d5a2f527d1915e22b00963af533e9d5da03"}
Nov 03 08:35:59.268866 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.268690    6430 scope.go:122] "RemoveContainer" containerID="56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7"
Nov 03 08:35:59.277462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.277436    6430 scope.go:122] "RemoveContainer" containerID="47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc"
Nov 03 08:35:59.285280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.285258    6430 scope.go:122] "RemoveContainer" containerID="56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7"
Nov 03 08:35:59.285556 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:35:59.285533    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7\": container with ID starting with 56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7 not found: ID does not exist" containerID="56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7"
Nov 03 08:35:59.285607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.285566    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7"} err="failed to get container status \"56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7\": rpc error: code = NotFound desc = could not find container \"56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7\": container with ID starting with 56142c8874f8eecc76a6983fdcccbe48b0bf2c2270d35711a14aa7687e11c6f7 not found: ID does not exist"
Nov 03 08:35:59.285607 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.285584    6430 scope.go:122] "RemoveContainer" containerID="47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc"
Nov 03 08:35:59.285797 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:35:59.285773    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc\": container with ID starting with 47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc not found: ID does not exist" containerID="47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc"
Nov 03 08:35:59.285838 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.285806    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc"} err="failed to get container status \"47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc\": rpc error: code = NotFound desc = could not find container \"47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc\": container with ID starting with 47862c867f07b07272c72e59b7cf110214517f9b165a07a48ca9452ec2687cdc not found: ID does not exist"
Nov 03 08:35:59.359350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359325    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-token\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-token\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359365    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-envoy\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359383    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-data\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359415    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-kube-api-access-mc2d5\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-kube-api-access-mc2d5\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359459 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359439    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-certs\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359471    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istiod-ca-cert\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359536    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-podinfo\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359590    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-socket\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359656    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-credential-socket\") pod \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\" (UID: \"6d7c573a-b82d-4bb3-bc71-ada5109471e3\") "
Nov 03 08:35:59.359703 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359657    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-data" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:59.359983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359763    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-certs" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:59.359983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359913    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-socket" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:59.359983 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.359952    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-credential-socket" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:59.360354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.360092    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.360354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.360113    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.360354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.360185    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.360354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.360294    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.360354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.360300    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istiod-ca-cert" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:35:59.361653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.361629    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-envoy" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:35:59.361766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.361751    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-token" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:35:59.361870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.361799    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-kube-api-access-mc2d5" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "kube-api-access-mc2d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:35:59.361870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.361814    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-podinfo" pod "6d7c573a-b82d-4bb3-bc71-ada5109471e3" (UID: "6d7c573a-b82d-4bb3-bc71-ada5109471e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:35:59.461875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.461840    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.462016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.461889    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.462016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.461904    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.462016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.461920    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc2d5\" (UniqueName: \"kubernetes.io/projected/6d7c573a-b82d-4bb3-bc71-ada5109471e3-kube-api-access-mc2d5\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.462016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.461936    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7c573a-b82d-4bb3-bc71-ada5109471e3-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.600701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.600668    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"]
Nov 03 08:35:59.611181 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:35:59.611150    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0af2f6-openshift-default-9fcd97b5c-j7fkt"]
Nov 03 08:36:00.615891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:00.615856    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" path="/var/lib/kubelet/pods/6d7c573a-b82d-4bb3-bc71-ada5109471e3/volumes"
Nov 03 08:36:00.829372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:00.829333    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" podUID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:36:00.829541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:00.829333    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" podUID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:36:10.829208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:10.829168    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" podUID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:36:10.829721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:10.829179    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" podUID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:36:12.300562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.300535    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:36:12.331818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.331791    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:36:12.379622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.379585    6430 generic.go:356] "Generic (PLEG): container finished" podID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" containerID="70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3" exitCode=0
Nov 03 08:36:12.379800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.379638    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"
Nov 03 08:36:12.379800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.379678    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" event={"ID":"9c1c5a17-29f1-4d82-8d2a-6750631d191a","Type":"ContainerDied","Data":"70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3"}
Nov 03 08:36:12.379800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.379733    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8" event={"ID":"9c1c5a17-29f1-4d82-8d2a-6750631d191a","Type":"ContainerDied","Data":"1586c8bd087665b509e359d9c995d233ca4ccca552d7e5498c9b42f11efd24b6"}
Nov 03 08:36:12.379800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.379763    6430 scope.go:122] "RemoveContainer" containerID="70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3"
Nov 03 08:36:12.381042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.381023    6430 generic.go:356] "Generic (PLEG): container finished" podID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" containerID="b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220" exitCode=0
Nov 03 08:36:12.381179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.381073    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"
Nov 03 08:36:12.381179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.381117    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" event={"ID":"8e33572d-a87b-4688-adf8-ed7f1b1944f3","Type":"ContainerDied","Data":"b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220"}
Nov 03 08:36:12.381179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.381166    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp" event={"ID":"8e33572d-a87b-4688-adf8-ed7f1b1944f3","Type":"ContainerDied","Data":"011c70a0d9597ada568a60cd9b2d4a41e643ea6c851ce558e00d581a0192cb21"}
Nov 03 08:36:12.388817 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.388787    6430 scope.go:122] "RemoveContainer" containerID="70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3"
Nov 03 08:36:12.389075 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:36:12.389043    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3\": container with ID starting with 70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3 not found: ID does not exist" containerID="70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3"
Nov 03 08:36:12.389134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.389086    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3"} err="failed to get container status \"70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3\": rpc error: code = NotFound desc = could not find container \"70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3\": container with ID starting with 70a4bcd612a5975470d858350b723d67fef3c6754b73d49220736d3895e661d3 not found: ID does not exist"
Nov 03 08:36:12.389134 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.389103    6430 scope.go:122] "RemoveContainer" containerID="b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220"
Nov 03 08:36:12.396856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.396841    6430 scope.go:122] "RemoveContainer" containerID="b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220"
Nov 03 08:36:12.397116 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:36:12.397094    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220\": container with ID starting with b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220 not found: ID does not exist" containerID="b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220"
Nov 03 08:36:12.397173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.397125    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220"} err="failed to get container status \"b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220\": rpc error: code = NotFound desc = could not find container \"b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220\": container with ID starting with b16516e2c819e3eac2f59002698fbabc9616c539380e199361cd9bbb47bd1220 not found: ID does not exist"
Nov 03 08:36:12.398352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.398336    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1c5a17-29f1-4d82-8d2a-6750631d191a-kube-api-access-zlfct\" (UniqueName: \"kubernetes.io/projected/9c1c5a17-29f1-4d82-8d2a-6750631d191a-kube-api-access-zlfct\") pod \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") "
Nov 03 08:36:12.398415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.398368    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e33572d-a87b-4688-adf8-ed7f1b1944f3-kube-api-access-99kg9\" (UniqueName: \"kubernetes.io/projected/8e33572d-a87b-4688-adf8-ed7f1b1944f3-kube-api-access-99kg9\") pod \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") "
Nov 03 08:36:12.398457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.398415    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e33572d-a87b-4688-adf8-ed7f1b1944f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e33572d-a87b-4688-adf8-ed7f1b1944f3-plugins-config-volume\") pod \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\" (UID: \"8e33572d-a87b-4688-adf8-ed7f1b1944f3\") "
Nov 03 08:36:12.398492 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.398464    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1c5a17-29f1-4d82-8d2a-6750631d191a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1c5a17-29f1-4d82-8d2a-6750631d191a-plugins-config-volume\") pod \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\" (UID: \"9c1c5a17-29f1-4d82-8d2a-6750631d191a\") "
Nov 03 08:36:12.398781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.398757    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e33572d-a87b-4688-adf8-ed7f1b1944f3-plugins-config-volume" pod "8e33572d-a87b-4688-adf8-ed7f1b1944f3" (UID: "8e33572d-a87b-4688-adf8-ed7f1b1944f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:36:12.398853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.398795    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1c5a17-29f1-4d82-8d2a-6750631d191a-plugins-config-volume" pod "9c1c5a17-29f1-4d82-8d2a-6750631d191a" (UID: "9c1c5a17-29f1-4d82-8d2a-6750631d191a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:36:12.400442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.400423    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1c5a17-29f1-4d82-8d2a-6750631d191a-kube-api-access-zlfct" pod "9c1c5a17-29f1-4d82-8d2a-6750631d191a" (UID: "9c1c5a17-29f1-4d82-8d2a-6750631d191a"). InnerVolumeSpecName "kube-api-access-zlfct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:36:12.400533 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.400516    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e33572d-a87b-4688-adf8-ed7f1b1944f3-kube-api-access-99kg9" pod "8e33572d-a87b-4688-adf8-ed7f1b1944f3" (UID: "8e33572d-a87b-4688-adf8-ed7f1b1944f3"). InnerVolumeSpecName "kube-api-access-99kg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:36:12.499214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.499179    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1c5a17-29f1-4d82-8d2a-6750631d191a-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:36:12.499214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.499207    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlfct\" (UniqueName: \"kubernetes.io/projected/9c1c5a17-29f1-4d82-8d2a-6750631d191a-kube-api-access-zlfct\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:36:12.499214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.499217    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99kg9\" (UniqueName: \"kubernetes.io/projected/8e33572d-a87b-4688-adf8-ed7f1b1944f3-kube-api-access-99kg9\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:36:12.499214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.499225    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e33572d-a87b-4688-adf8-ed7f1b1944f3-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:36:12.695365 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.695334    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"]
Nov 03 08:36:12.697891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.697868    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0af2f6-7dcc787ffclfx8"]
Nov 03 08:36:12.706156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.706131    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"]
Nov 03 08:36:12.708326 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.708307    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0af2f6-6b899nndqp"]
Nov 03 08:36:12.710468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:12.710448    6430 scope.go:122] "RemoveContainer" containerID="0588a697d35d06e907d07de29414486e73d0f3e1e31010408dcda01ad4e29771"
Nov 03 08:36:14.615016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:14.614981    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" path="/var/lib/kubelet/pods/8e33572d-a87b-4688-adf8-ed7f1b1944f3/volumes"
Nov 03 08:36:14.615406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:14.615353    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" path="/var/lib/kubelet/pods/9c1c5a17-29f1-4d82-8d2a-6750631d191a/volumes"
Nov 03 08:36:14.847386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:14.847355    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:36:18.204215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:18.204178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:36:19.687719 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:19.687636    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:36:22.019031 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:22.018991    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"]
Nov 03 08:36:22.020556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:22.020529    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c2kgf"]
Nov 03 08:36:22.615034 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:22.615005    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a89702-510f-436c-8443-b1eba9764b6d" path="/var/lib/kubelet/pods/91a89702-510f-436c-8443-b1eba9764b6d/volumes"
Nov 03 08:36:25.117671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:25.117637    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:36:28.373858 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:36:28.373819    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:36:32.020748 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:32.020713    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"]
Nov 03 08:36:32.023207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:32.023181    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d2rgw"]
Nov 03 08:36:32.615039 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:32.615001    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcd7cc6-4535-4e88-a48b-ea9165199457" path="/var/lib/kubelet/pods/afcd7cc6-4535-4e88-a48b-ea9165199457/volumes"
Nov 03 08:36:53.021380 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:53.021348    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"]
Nov 03 08:36:53.025176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:53.025149    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdsvs"]
Nov 03 08:36:54.615118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:54.615081    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc857b2e-f99f-4400-852b-7f7b1fb9639b" path="/var/lib/kubelet/pods/fc857b2e-f99f-4400-852b-7f7b1fb9639b/volumes"
Nov 03 08:36:55.786080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:36:55.786020    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:36:58.374534 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:36:58.374497    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:37:11.019377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:11.019343    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"]
Nov 03 08:37:11.021900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:11.021874    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qkwx6"]
Nov 03 08:37:12.615164 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:12.615134    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d53509-062d-4e90-8f81-c89bcdef9010" path="/var/lib/kubelet/pods/86d53509-062d-4e90-8f81-c89bcdef9010/volumes"
Nov 03 08:37:12.749728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:12.749696    6430 scope.go:122] "RemoveContainer" containerID="a10267f84b88575e1f6a043a87b0401d222370c208fd42cdb534bdef2ec9f128"
Nov 03 08:37:12.757997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:12.757976    6430 scope.go:122] "RemoveContainer" containerID="22dc61e2fc52e405d2384f8d388c2ca6b9da962ad27dddeee4b2832b83538d87"
Nov 03 08:37:12.765692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:12.765676    6430 scope.go:122] "RemoveContainer" containerID="2d2e79a7c8889eb4fe052026f196adfe28065eb1ce0765bba036bfbc5bf9b80d"
Nov 03 08:37:14.887953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:14.887916    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:37:15.840105 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:15.840045    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:37:28.374105 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:37:28.374037    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:37:30.021002 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:30.020966    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"]
Nov 03 08:37:30.025860 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:30.025832    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jmw6q"]
Nov 03 08:37:30.615892 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:30.615858    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb" path="/var/lib/kubelet/pods/0d1fa25e-d6d1-4ae5-b37c-95daaf2cdffb/volumes"
Nov 03 08:37:40.412628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.412582    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"]
Nov 03 08:37:40.413178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413103    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" containerName="payload-processing"
Nov 03 08:37:40.413178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413122    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" containerName="payload-processing"
Nov 03 08:37:40.413178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413139    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e9f46534-edb9-4ee5-87cf-10853efcb7c2" containerName="maas-api"
Nov 03 08:37:40.413178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413146    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f46534-edb9-4ee5-87cf-10853efcb7c2" containerName="maas-api"
Nov 03 08:37:40.413178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413169    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" containerName="payload-pre-processing"
Nov 03 08:37:40.413178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413177    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" containerName="payload-pre-processing"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413213    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413221    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413231    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5b8ac031-b54e-4a47-b0ad-01f9faec501e" containerName="main"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413239    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8ac031-b54e-4a47-b0ad-01f9faec501e" containerName="main"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413251    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413259    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413354    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413369    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8ac031-b54e-4a47-b0ad-01f9faec501e" containerName="main"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413380    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7c573a-b82d-4bb3-bc71-ada5109471e3" containerName="istio-proxy"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413390    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1c5a17-29f1-4d82-8d2a-6750631d191a" containerName="payload-processing"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413399    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f46534-edb9-4ee5-87cf-10853efcb7c2" containerName="maas-api"
Nov 03 08:37:40.413526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.413410    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e33572d-a87b-4688-adf8-ed7f1b1944f3" containerName="payload-pre-processing"
Nov 03 08:37:40.416525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.416504    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.419634 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.419424    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:37:40.420507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.420479    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:37:40.427213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.427005    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"]
Nov 03 08:37:40.590584 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.590550    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8nk\" (UniqueName: \"kubernetes.io/projected/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-kube-api-access-bh8nk\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.590765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.590631    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.590765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.590670    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.691437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.691411    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8nk\" (UniqueName: \"kubernetes.io/projected/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-kube-api-access-bh8nk\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.691615 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.691461    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.691615 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.691486    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.692054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.692036    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.693945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.693924    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.698390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.698364    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8nk\" (UniqueName: \"kubernetes.io/projected/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-kube-api-access-bh8nk\") pod \"maas-api-revoke-sub-d576e68b-l8zz8\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:40.738368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:40.738344    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:41.009946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:41.009875    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"]
Nov 03 08:37:41.013318 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:37:41.013282    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc796c424_6c1a_43b1_bd5e_ad9e9cd76d9a.slice/crio-6ed82614189333d37b2c3c7d99579fabf79cb0ac7949abff78dfd2039d6e3fe3 WatchSource:0}: Error finding container 6ed82614189333d37b2c3c7d99579fabf79cb0ac7949abff78dfd2039d6e3fe3: Status 404 returned error can't find the container with id 6ed82614189333d37b2c3c7d99579fabf79cb0ac7949abff78dfd2039d6e3fe3
Nov 03 08:37:41.796949 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:41.796917    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8" event={"ID":"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a","Type":"ContainerStarted","Data":"7863ca21b36598886a3032f8ff72c94f95fc2c2fef5790bb88a43f2c932f7f28"}
Nov 03 08:37:41.797412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:41.796971    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8" event={"ID":"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a","Type":"ContainerStarted","Data":"6ed82614189333d37b2c3c7d99579fabf79cb0ac7949abff78dfd2039d6e3fe3"}
Nov 03 08:37:41.810862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:41.810814    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8" podStartSLOduration=1.8108013729999999 podStartE2EDuration="1.810801373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:37:41.810751947 +0000 UTC m=+2611.313587463" watchObservedRunningTime="2026-11-03 08:37:41.810801373 +0000 UTC m=+2611.313636887"
Nov 03 08:37:42.803210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:42.803172    6430 generic.go:356] "Generic (PLEG): container finished" podID="c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" containerID="7863ca21b36598886a3032f8ff72c94f95fc2c2fef5790bb88a43f2c932f7f28" exitCode=0
Nov 03 08:37:42.803642 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:42.803255    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8" event={"ID":"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a","Type":"ContainerDied","Data":"7863ca21b36598886a3032f8ff72c94f95fc2c2fef5790bb88a43f2c932f7f28"}
Nov 03 08:37:43.942471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:43.942444    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:44.023800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.023761    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-openshift-ca\") pod \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") "
Nov 03 08:37:44.024019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.023816    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-kube-api-access-bh8nk\" (UniqueName: \"kubernetes.io/projected/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-kube-api-access-bh8nk\") pod \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") "
Nov 03 08:37:44.024019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.023897    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-xks-ca\") pod \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\" (UID: \"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a\") "
Nov 03 08:37:44.024229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.024201    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-openshift-ca" pod "c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" (UID: "c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:37:44.026209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.026181    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-xks-ca" pod "c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" (UID: "c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:37:44.026341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.026211    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-kube-api-access-bh8nk" pod "c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" (UID: "c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a"). InnerVolumeSpecName "kube-api-access-bh8nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:37:44.125019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.124924    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:37:44.125019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.124962    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:37:44.125019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.124973    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh8nk\" (UniqueName: \"kubernetes.io/projected/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a-kube-api-access-bh8nk\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:37:44.813708 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.813677    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"
Nov 03 08:37:44.813901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.813670    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8" event={"ID":"c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a","Type":"ContainerDied","Data":"6ed82614189333d37b2c3c7d99579fabf79cb0ac7949abff78dfd2039d6e3fe3"}
Nov 03 08:37:44.813901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:44.813801    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed82614189333d37b2c3c7d99579fabf79cb0ac7949abff78dfd2039d6e3fe3"
Nov 03 08:37:47.020853 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:47.020819    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"]
Nov 03 08:37:47.023284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:47.023253    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n86sq"]
Nov 03 08:37:48.614903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:48.614872    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161a6a4e-79d1-45fe-a53b-cf2cf6f151da" path="/var/lib/kubelet/pods/161a6a4e-79d1-45fe-a53b-cf2cf6f151da/volumes"
Nov 03 08:37:58.373937 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:37:58.373851    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:37:59.497941 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:37:59.497904    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:01.185256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.185216    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"]
Nov 03 08:38:01.185671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.185578    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" containerName="revoke-keys"
Nov 03 08:38:01.185671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.185589    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" containerName="revoke-keys"
Nov 03 08:38:01.185747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.185683    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" containerName="revoke-keys"
Nov 03 08:38:01.188777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.188755    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.192302 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.192279    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:38:01.192427 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.192280    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:38:01.199996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.199975    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"]
Nov 03 08:38:01.301412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.301372    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67llp\" (UniqueName: \"kubernetes.io/projected/788c88e6-6590-4317-b1db-922f79ff46af-kube-api-access-67llp\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.301602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.301446    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.301602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.301547    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.402856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.402811    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67llp\" (UniqueName: \"kubernetes.io/projected/788c88e6-6590-4317-b1db-922f79ff46af-kube-api-access-67llp\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.403015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.402877    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.403015 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.402900    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.403568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.403548    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.405540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.405520    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.411471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.411448    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67llp\" (UniqueName: \"kubernetes.io/projected/788c88e6-6590-4317-b1db-922f79ff46af-kube-api-access-67llp\") pod \"maas-api-revoke-sub-d2bc93bc-vlfkx\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.501646 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.501564    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:01.775097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.775002    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"]
Nov 03 08:38:01.778329 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:38:01.778296    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788c88e6_6590_4317_b1db_922f79ff46af.slice/crio-1eaa7ef5816accb7668b226c703633d633766e70f4fbfb6d83bd167dd09bd33a WatchSource:0}: Error finding container 1eaa7ef5816accb7668b226c703633d633766e70f4fbfb6d83bd167dd09bd33a: Status 404 returned error can't find the container with id 1eaa7ef5816accb7668b226c703633d633766e70f4fbfb6d83bd167dd09bd33a
Nov 03 08:38:01.906514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:01.906469    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx" event={"ID":"788c88e6-6590-4317-b1db-922f79ff46af","Type":"ContainerStarted","Data":"1eaa7ef5816accb7668b226c703633d633766e70f4fbfb6d83bd167dd09bd33a"}
Nov 03 08:38:02.919067 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:02.919032    6430 generic.go:356] "Generic (PLEG): container finished" podID="788c88e6-6590-4317-b1db-922f79ff46af" containerID="53b9e66380b44ad0eec2a89555a8f9a7dae4db93b86bbf5f252c80987b179ff8" exitCode=0
Nov 03 08:38:02.919438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:02.919121    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx" event={"ID":"788c88e6-6590-4317-b1db-922f79ff46af","Type":"ContainerDied","Data":"53b9e66380b44ad0eec2a89555a8f9a7dae4db93b86bbf5f252c80987b179ff8"}
Nov 03 08:38:04.053206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.053185    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:04.129681 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.129654    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788c88e6-6590-4317-b1db-922f79ff46af-kube-api-access-67llp\" (UniqueName: \"kubernetes.io/projected/788c88e6-6590-4317-b1db-922f79ff46af-kube-api-access-67llp\") pod \"788c88e6-6590-4317-b1db-922f79ff46af\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") "
Nov 03 08:38:04.129828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.129689    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-openshift-ca\") pod \"788c88e6-6590-4317-b1db-922f79ff46af\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") "
Nov 03 08:38:04.129828 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.129773    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-xks-ca\") pod \"788c88e6-6590-4317-b1db-922f79ff46af\" (UID: \"788c88e6-6590-4317-b1db-922f79ff46af\") "
Nov 03 08:38:04.130124 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.130100    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-openshift-ca" pod "788c88e6-6590-4317-b1db-922f79ff46af" (UID: "788c88e6-6590-4317-b1db-922f79ff46af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:38:04.131863 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.131837    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-xks-ca" pod "788c88e6-6590-4317-b1db-922f79ff46af" (UID: "788c88e6-6590-4317-b1db-922f79ff46af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:38:04.131942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.131858    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788c88e6-6590-4317-b1db-922f79ff46af-kube-api-access-67llp" pod "788c88e6-6590-4317-b1db-922f79ff46af" (UID: "788c88e6-6590-4317-b1db-922f79ff46af"). InnerVolumeSpecName "kube-api-access-67llp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:38:04.231048 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.231017    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:38:04.231048 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.231045    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67llp\" (UniqueName: \"kubernetes.io/projected/788c88e6-6590-4317-b1db-922f79ff46af-kube-api-access-67llp\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:38:04.231227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.231076    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788c88e6-6590-4317-b1db-922f79ff46af-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:38:04.929682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.929593    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx" event={"ID":"788c88e6-6590-4317-b1db-922f79ff46af","Type":"ContainerDied","Data":"1eaa7ef5816accb7668b226c703633d633766e70f4fbfb6d83bd167dd09bd33a"}
Nov 03 08:38:04.929682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.929633    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"
Nov 03 08:38:04.929938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:04.929637    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eaa7ef5816accb7668b226c703633d633766e70f4fbfb6d83bd167dd09bd33a"
Nov 03 08:38:07.678375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:07.678341    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:07.795770 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:07.795735    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:12.595947 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.595913    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:12.793374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.793343    6430 scope.go:122] "RemoveContainer" containerID="408ede2b5d38792056fc12fb2d389fe33b90b3f2e10301e1638272b5a674883a"
Nov 03 08:38:12.801745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.801726    6430 scope.go:122] "RemoveContainer" containerID="62ff32b41cfd4662f359e94caf7b3ab139825077e68547fd214ee87630d928c0"
Nov 03 08:38:12.809483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.809455    6430 scope.go:122] "RemoveContainer" containerID="5697679ef1c760be0e6a648b36aa8cdb906333c9ec4267465722f1a90d53a483"
Nov 03 08:38:12.816717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.816697    6430 scope.go:122] "RemoveContainer" containerID="07e075667bee3f68ff921a4e202f662227b5a644d9af83b9fb3d6da8b92ae752"
Nov 03 08:38:12.921884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.920712    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"]
Nov 03 08:38:12.921884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.921249    6430 cpu_manager.go:414] "containerMap: removing container" podUID="788c88e6-6590-4317-b1db-922f79ff46af" containerName="revoke-keys"
Nov 03 08:38:12.921884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.921266    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788c88e6-6590-4317-b1db-922f79ff46af" containerName="revoke-keys"
Nov 03 08:38:12.921884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.921391    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="788c88e6-6590-4317-b1db-922f79ff46af" containerName="revoke-keys"
Nov 03 08:38:12.926437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.926413    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:12.934898 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.934644    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-890648-openshift-default-dockercfg-922j4\""
Nov 03 08:38:12.936809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.936787    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"]
Nov 03 08:38:12.988439 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:12.988404    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:13.002011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.001976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istiod-ca-cert\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002010    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-data\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002031    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-credential-socket\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002127    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-certs\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002221    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-socket\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002243    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-token\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002283    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-envoy\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002366    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-podinfo\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.002501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.002478    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82f6\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-kube-api-access-n82f6\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.103835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103731    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-certs\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.103835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103797    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-socket\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.103835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103816    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-token\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103847    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-envoy\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103874    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-podinfo\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103892    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82f6\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-kube-api-access-n82f6\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103932    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istiod-ca-cert\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103956    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-data\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104183 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.103989    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-credential-socket\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.104257    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-certs\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.104305    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-socket\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.104632    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-data\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104730 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.104672    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-credential-socket\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.104851 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.104833    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istiod-ca-cert\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.106925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.106893    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-envoy\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.107041 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.107003    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-podinfo\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.112120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.112091    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-token\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.112404 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.112387    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82f6\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-kube-api-access-n82f6\") pod \"e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") " pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.235279 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.235191    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:13.242361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.242325    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:13.522373 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.522338    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"]
Nov 03 08:38:13.526179 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:38:13.526148    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c14b18c_6836_4ed5_b22d_ab4fa98bf9e2.slice/crio-0eef07118288581563f0214a549fc1542c67384ac9e0e2f8555e104b4d3f2d5a WatchSource:0}: Error finding container 0eef07118288581563f0214a549fc1542c67384ac9e0e2f8555e104b4d3f2d5a: Status 404 returned error can't find the container with id 0eef07118288581563f0214a549fc1542c67384ac9e0e2f8555e104b4d3f2d5a
Nov 03 08:38:13.528406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.528371    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:38:13.528493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.528438    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:38:13.528493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.528470    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:38:13.975038 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.975007    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" event={"ID":"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2","Type":"ContainerStarted","Data":"c10ca23e794186311e4372c102f388f1736ba35d82cb20354635344e3138920f"}
Nov 03 08:38:13.975540 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.975084    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" event={"ID":"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2","Type":"ContainerStarted","Data":"0eef07118288581563f0214a549fc1542c67384ac9e0e2f8555e104b4d3f2d5a"}
Nov 03 08:38:13.994284 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:13.994207    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podStartSLOduration=1.994194918 podStartE2EDuration="1.994194918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:13.99279707 +0000 UTC m=+2643.495632585" watchObservedRunningTime="2026-11-03 08:38:13.994194918 +0000 UTC m=+2643.497030431"
Nov 03 08:38:14.242663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:14.242575    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:14.247875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:14.247846    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:14.940293 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:14.940260    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:14.979773 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:14.979746    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:14.980648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:14.980631    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:16.224417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:16.224360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:17.269503 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.269469    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"]
Nov 03 08:38:17.273126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.273103    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.278200 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.278175    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-890648-serving-cert\""
Nov 03 08:38:17.278351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.278262    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 08:38:17.278489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.278472    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7nz66\""
Nov 03 08:38:17.283571 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.283550    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"]
Nov 03 08:38:17.338706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.338675    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.338706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.338708    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.338911 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.338739    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.338948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.338912    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cq58\" (UniqueName: \"kubernetes.io/projected/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-kube-api-access-4cq58\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.440290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.440255    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.440460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.440313    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq58\" (UniqueName: \"kubernetes.io/projected/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-kube-api-access-4cq58\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.440460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.440377    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.440460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.440397    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.441022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.440988    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.442970 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.442941    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.443106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.442955    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.448364 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.448339    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"]
Nov 03 08:38:17.451921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.451904    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.453725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.453702    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq58\" (UniqueName: \"kubernetes.io/projected/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-kube-api-access-4cq58\") pod \"maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.459089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.459044    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-890648\""
Nov 03 08:38:17.460765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.460746    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"]
Nov 03 08:38:17.481909 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.481886    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"]
Nov 03 08:38:17.485908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.485888    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.485998 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.485979    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:17.495852 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.495832    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-276d8f54-dockercfg-vbnlw\""
Nov 03 08:38:17.497490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.497469    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"]
Nov 03 08:38:17.540974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.540896    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqm6\" (UniqueName: \"kubernetes.io/projected/6aeed716-8861-4588-b7fe-a33508429567-kube-api-access-fnqm6\") pod \"payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.540974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.540956    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aeed716-8861-4588-b7fe-a33508429567-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.588537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.588502    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:17.641722 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.641681    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqm6\" (UniqueName: \"kubernetes.io/projected/6aeed716-8861-4588-b7fe-a33508429567-kube-api-access-fnqm6\") pod \"payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.641903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.641765    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpsn\" (UniqueName: \"kubernetes.io/projected/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-kube-api-access-kkpsn\") pod \"payload-pre-processing-e2e-worker-single-main-890648-8545496m67\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.641903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.641797    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aeed716-8861-4588-b7fe-a33508429567-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.641903 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.641888    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-890648-8545496m67\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.642656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.642629    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aeed716-8861-4588-b7fe-a33508429567-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.650955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.650931    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqm6\" (UniqueName: \"kubernetes.io/projected/6aeed716-8861-4588-b7fe-a33508429567-kube-api-access-fnqm6\") pod \"payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.743186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.743157    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpsn\" (UniqueName: \"kubernetes.io/projected/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-kube-api-access-kkpsn\") pod \"payload-pre-processing-e2e-worker-single-main-890648-8545496m67\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.743351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.743207    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-890648-8545496m67\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.743869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.743849    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-890648-8545496m67\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.753354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.753329    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpsn\" (UniqueName: \"kubernetes.io/projected/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-kube-api-access-kkpsn\") pod \"payload-pre-processing-e2e-worker-single-main-890648-8545496m67\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.778269 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.778232    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:17.799553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.799532    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:17.878226 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.878147    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"]
Nov 03 08:38:17.882393 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:38:17.882348    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed821a6_7a8b_44ec_b448_fa8c1702e6a2.slice/crio-9b25109cb640e3e0c3fb670a65ae8c9bbb786cc668f417d1b9ca07fb0e45637d WatchSource:0}: Error finding container 9b25109cb640e3e0c3fb670a65ae8c9bbb786cc668f417d1b9ca07fb0e45637d: Status 404 returned error can't find the container with id 9b25109cb640e3e0c3fb670a65ae8c9bbb786cc668f417d1b9ca07fb0e45637d
Nov 03 08:38:17.892940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.892173    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:38:17.996900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:17.996861    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99" event={"ID":"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2","Type":"ContainerStarted","Data":"9b25109cb640e3e0c3fb670a65ae8c9bbb786cc668f417d1b9ca07fb0e45637d"}
Nov 03 08:38:18.055476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:18.055403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:18.059206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:18.059139    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"]
Nov 03 08:38:18.061860 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:38:18.061835    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeed716_8861_4588_b7fe_a33508429567.slice/crio-c36bb7c5c638146b36d9d00cdf1cb4a28193b404199ff7fc7a28c89dc310cc3c WatchSource:0}: Error finding container c36bb7c5c638146b36d9d00cdf1cb4a28193b404199ff7fc7a28c89dc310cc3c: Status 404 returned error can't find the container with id c36bb7c5c638146b36d9d00cdf1cb4a28193b404199ff7fc7a28c89dc310cc3c
Nov 03 08:38:18.087254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:18.087222    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"]
Nov 03 08:38:18.093307 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:38:18.093278    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ec5be7_cd3b_44bf_b1d2_731b0ddced4a.slice/crio-62036cf56033f79fa908a3f7a6c7f570abb7a187473313ecb117df7a38f1ff71 WatchSource:0}: Error finding container 62036cf56033f79fa908a3f7a6c7f570abb7a187473313ecb117df7a38f1ff71: Status 404 returned error can't find the container with id 62036cf56033f79fa908a3f7a6c7f570abb7a187473313ecb117df7a38f1ff71
Nov 03 08:38:18.653249 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:18.653219    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:19.005812 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.005767    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99" event={"ID":"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2","Type":"ContainerStarted","Data":"7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266"}
Nov 03 08:38:19.005984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.005945    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:19.007448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.007420    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" event={"ID":"6aeed716-8861-4588-b7fe-a33508429567","Type":"ContainerStarted","Data":"b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13"}
Nov 03 08:38:19.007568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.007473    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:19.007568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.007497    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" event={"ID":"6aeed716-8861-4588-b7fe-a33508429567","Type":"ContainerStarted","Data":"c36bb7c5c638146b36d9d00cdf1cb4a28193b404199ff7fc7a28c89dc310cc3c"}
Nov 03 08:38:19.008925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.008907    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" event={"ID":"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a","Type":"ContainerStarted","Data":"a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0"}
Nov 03 08:38:19.008997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.008938    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" event={"ID":"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a","Type":"ContainerStarted","Data":"62036cf56033f79fa908a3f7a6c7f570abb7a187473313ecb117df7a38f1ff71"}
Nov 03 08:38:19.009072 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.009042    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:19.024556 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.024513    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99" podStartSLOduration=1.485368637 podStartE2EDuration="2.02449921s" totalImagesPullingTime="539.130573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:19.022747118 +0000 UTC m=+2648.525582632" watchObservedRunningTime="2026-11-03 08:38:19.02449921 +0000 UTC m=+2648.527334749"
Nov 03 08:38:19.038689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.038646    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" podStartSLOduration=2.038631251 podStartE2EDuration="2.038631251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:19.037295276 +0000 UTC m=+2648.540130791" watchObservedRunningTime="2026-11-03 08:38:19.038631251 +0000 UTC m=+2648.541466764"
Nov 03 08:38:19.053638 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.053602    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" podStartSLOduration=2.05359202 podStartE2EDuration="2.05359202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:19.053253508 +0000 UTC m=+2648.556089023" watchObservedRunningTime="2026-11-03 08:38:19.05359202 +0000 UTC m=+2648.556427531"
Nov 03 08:38:19.255408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.255363    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:19.903949 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:19.903911    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:20.557203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:20.557167    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:21.222093 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:21.221772    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:21.941928 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:21.941873    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:22.456080 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:22.456023    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:23.049567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:23.049521    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:23.732489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:23.732453    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:24.517087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:24.516685    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:25.028514 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:25.028480    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:38:25.261372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:25.261332    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:26.021960 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:26.021928    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:26.749785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:26.749747    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:27.433422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:27.433384    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:28.034619 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:28.034583    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:28.377716 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:38:28.377620    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:38:28.705430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:28.705399    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:29.071406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.071320    6430 generic.go:356] "Generic (PLEG): container finished" podID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerID="c10ca23e794186311e4372c102f388f1736ba35d82cb20354635344e3138920f" exitCode=0
Nov 03 08:38:29.071406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.071369    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" event={"ID":"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2","Type":"ContainerDied","Data":"c10ca23e794186311e4372c102f388f1736ba35d82cb20354635344e3138920f"}
Nov 03 08:38:29.071856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.071727    6430 scope.go:122] "RemoveContainer" containerID="c10ca23e794186311e4372c102f388f1736ba35d82cb20354635344e3138920f"
Nov 03 08:38:29.072789 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.072759    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:38:29.072904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.072854    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:38:29.072974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.072910    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:38:29.251275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.251234    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:29.867056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.867027    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:29.979811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:29.979776    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:30.024182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:30.024141    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:38:30.024732 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:30.024697    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:38:30.078922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:30.078892    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" event={"ID":"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2","Type":"ContainerStarted","Data":"0c5052247d734162aae16c0169e72b945b0aa99f92d1733588512abca38f3b25"}
Nov 03 08:38:30.242966 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:30.242923    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:30.244211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:30.244182    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:38:30.244330 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:30.244266    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 08:38:30.561161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:30.561095    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:31.143671 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:31.143631    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:31.243122 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:31.243053    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:38:31.243314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:31.243137    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 08:38:31.354562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:31.354523    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:31.760410 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:31.760375    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:31.890461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:31.890431    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:32.059381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.059302    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:32.242836 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.242795    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:38:32.243263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.242870    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 08:38:32.866429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.866393    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:32.913572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.913513    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"]
Nov 03 08:38:32.941792 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.941750    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"]
Nov 03 08:38:32.942092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.941990    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:32.947507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.947482    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-890648\"/\"openshift-service-ca.crt\""
Nov 03 08:38:32.947662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.947500    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-890648\"/\"kube-root-ca.crt\""
Nov 03 08:38:32.952375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.952351    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-890648\"/\"default-dockercfg-l4phc\""
Nov 03 08:38:32.952723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.952684    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-890648\"/\"simulator-single-main-890648-kserve-self-signed-certs\""
Nov 03 08:38:32.983207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:32.983172    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:33.037826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.037788    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-home\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.037826 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.037834    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tmp-dir\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.038081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.037963    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tls-certs\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.038081 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.038021    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-998tt\" (UniqueName: \"kubernetes.io/projected/32bbf3ea-6531-4c5a-9509-05ec34940e7a-kube-api-access-998tt\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.038176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.038150    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-model-cache\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.038215 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.038187    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-dshm\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.139457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139376    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-home\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.139457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139419    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tmp-dir\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.139727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139471    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tls-certs\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.139727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139505    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-998tt\" (UniqueName: \"kubernetes.io/projected/32bbf3ea-6531-4c5a-9509-05ec34940e7a-kube-api-access-998tt\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.139727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139577    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-model-cache\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.139727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139618    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-dshm\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.139975 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139926    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-home\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.140109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.139969    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tmp-dir\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.140109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.140033    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-model-cache\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.142011 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.141990    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-dshm\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.142466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.142443    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tls-certs\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.146602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.146577    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-998tt\" (UniqueName: \"kubernetes.io/projected/32bbf3ea-6531-4c5a-9509-05ec34940e7a-kube-api-access-998tt\") pod \"simulator-single-main-890648-kserve-7b695c7599-pnvk5\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") " pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.243126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.243087    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:38:33.243570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.243146    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 08:38:33.257875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.257846    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:33.567585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.567551    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"]
Nov 03 08:38:33.570664 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:33.570634    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:33.572097 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:38:33.572038    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32bbf3ea_6531_4c5a_9509_05ec34940e7a.slice/crio-bf802261d67275c24cd88d0b8a85b3ceb60db041a212324c0535a22c5cb71ff2 WatchSource:0}: Error finding container bf802261d67275c24cd88d0b8a85b3ceb60db041a212324c0535a22c5cb71ff2: Status 404 returned error can't find the container with id bf802261d67275c24cd88d0b8a85b3ceb60db041a212324c0535a22c5cb71ff2
Nov 03 08:38:34.140323 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.140284    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5" event={"ID":"32bbf3ea-6531-4c5a-9509-05ec34940e7a","Type":"ContainerStarted","Data":"64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2"}
Nov 03 08:38:34.140502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.140332    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5" event={"ID":"32bbf3ea-6531-4c5a-9509-05ec34940e7a","Type":"ContainerStarted","Data":"bf802261d67275c24cd88d0b8a85b3ceb60db041a212324c0535a22c5cb71ff2"}
Nov 03 08:38:34.156961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.156916    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5" podStartSLOduration=2.156904065 podStartE2EDuration="2.156904065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:34.155801418 +0000 UTC m=+2663.658636932" watchObservedRunningTime="2026-11-03 08:38:34.156904065 +0000 UTC m=+2663.659739623"
Nov 03 08:38:34.158522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.158499    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:34.243684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.243647    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:38:34.244100 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.243701    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 08:38:34.258414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.258378    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:34.867114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:34.867081    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:35.626374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:35.626343    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:35.626830 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:35.626612    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:35.627394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:35.627373    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:38:35.650401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:35.650379    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:36.355385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:36.355353    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:36.940617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:36.940586    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:37.654647 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:37.654611    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:38.332457 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:38.332421    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:38.939321 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:38.939292    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:39.528793 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:39.528765    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:40.131497 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:40.131466    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:40.738159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:40.738128    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:41.330731 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:41.330698    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:41.958535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:41.958501    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:42.539553 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:42.539517    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:43.133484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:43.133448    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:43.258460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:43.258424    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:43.271153 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:43.271122    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:43.738411 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:43.738377    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:44.200296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:44.200264    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:38:44.360461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:44.360431    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:45.041541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:45.041503    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:45.653821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:45.653785    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:46.241864 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:46.241829    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:46.849811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:46.849778    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:47.471811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:47.471778    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:48.163721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:48.163686    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:49.034862 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:49.034804    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:49.640591 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:49.640559    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:50.018880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:50.018851    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"]
Nov 03 08:38:50.021129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:50.021102    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-whxsg"]
Nov 03 08:38:50.229328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:50.229296    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:50.615685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:50.615646    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dba73a-8e41-4962-a36a-d14bd5d6af61" path="/var/lib/kubelet/pods/d5dba73a-8e41-4962-a36a-d14bd5d6af61/volumes"
Nov 03 08:38:50.852403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:50.852366    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:51.654931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:51.654894    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:52.299583 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:52.299550    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:52.852437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:52.852403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:53.442126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:53.442055    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:54.045930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:54.045896    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:54.643113 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:54.643056    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:55.253232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:55.253199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:55.831701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:55.831672    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:56.464656 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:56.464617    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:57.050334 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:57.050299    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:57.651924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:57.651877    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:58.261693 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:58.261654    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:58.374407 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:38:58.374368    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:38:58.982501 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:58.982462    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:38:59.673400 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:38:59.673367    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:00.263872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:00.263831    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:00.990089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:00.990025    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:01.021030 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:01.021007    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"]
Nov 03 08:39:01.025300 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:01.025278    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s8vqc"]
Nov 03 08:39:01.659272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:01.659239    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:02.246256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:02.246218    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:02.615446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:02.615362    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b125a5a-4ee8-44c7-9b7a-6cb836170fad" path="/var/lib/kubelet/pods/8b125a5a-4ee8-44c7-9b7a-6cb836170fad/volumes"
Nov 03 08:39:02.931655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:02.931631    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:03.523614 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:03.523549    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:04.232250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:04.232199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:04.843460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:04.843408    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:05.450372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:05.450334    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:06.039993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:06.039956    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:06.658152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:06.658098    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:07.302697 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:07.302668    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:07.843162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:07.843128    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:08.441555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:08.441503    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:09.031422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:09.031382    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:09.618274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:09.618235    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:10.228769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:10.228733    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:10.840873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:10.840845    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:11.433932 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:11.433900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:12.133265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:12.133210    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:12.725054 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:12.725006    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:12.852131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:12.852104    6430 scope.go:122] "RemoveContainer" containerID="8ea6f546523209af61e6124dab2b695e1cbe0e168083e7ec08d3d48ad5c01720"
Nov 03 08:39:12.860529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:12.860510    6430 scope.go:122] "RemoveContainer" containerID="be2a3947575f15eb7bbbbd6059a3062b7ce7ec431e6e6577e73fda26dacff6e0"
Nov 03 08:39:12.868507 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:12.868489    6430 scope.go:122] "RemoveContainer" containerID="38e6dd0cc4cfd38484adc33c31e4a8b0d93008c334b3e56bd9077fd1a25a4ba5"
Nov 03 08:39:13.332663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:13.332628    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:13.616843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:13.616719    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:14.146954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:14.146918    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:14.993570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:14.993521    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:15.119097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:15.119051    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:15.646177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:15.646139    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:16.249984 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:16.249948    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:16.937182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:16.937131    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:17.569650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:17.569610    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:18.145205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:18.145167    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:18.546421 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:18.546390    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"]
Nov 03 08:39:18.642908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:18.642873    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:18.755845 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:18.755808    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:19.351476 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:19.351390    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:20.044408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:20.044375    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:20.416663 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:20.416575    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:20.544396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:20.544365    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:21.158432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:21.158399    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:21.769106 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:21.769052    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:22.347146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:22.347110    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:23.042622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:23.042589    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:23.678222 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:23.678186    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:23.848954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:23.848922    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"]
Nov 03 08:39:23.868135 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:23.868100    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"]
Nov 03 08:39:23.947572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:23.947533    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:24.076296 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.076268    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:24.101741 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.101702    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"]
Nov 03 08:39:24.102019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.101976    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99" podUID="7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" containerName="maas-api" containerID="cri-o://7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266" gracePeriod=30
Nov 03 08:39:24.197526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.197466    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:24.321672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.321588    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:24.444956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.444922    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:24.556876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.556851    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:39:24.638669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.638587    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-metrics-certs\") pod \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") "
Nov 03 08:39:24.638669 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.638657    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-maas-api-tls\") pod \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") "
Nov 03 08:39:24.638921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.638685    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-openshift-service-ca\") pod \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") "
Nov 03 08:39:24.638921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.638750    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-kube-api-access-4cq58\" (UniqueName: \"kubernetes.io/projected/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-kube-api-access-4cq58\") pod \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\" (UID: \"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2\") "
Nov 03 08:39:24.639098 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.639039    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-openshift-service-ca" pod "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" (UID: "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:39:24.640945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.640919    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-metrics-certs" pod "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" (UID: "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:39:24.640945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.640941    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-maas-api-tls" pod "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" (UID: "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:39:24.641186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.640989    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-kube-api-access-4cq58" pod "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" (UID: "7ed821a6-7a8b-44ec-b448-fa8c1702e6a2"). InnerVolumeSpecName "kube-api-access-4cq58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:24.740295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.740262    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:24.740295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.740293    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:24.740674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.740304    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:24.740674 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.740314    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cq58\" (UniqueName: \"kubernetes.io/projected/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2-kube-api-access-4cq58\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:24.939107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:24.939051    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:25.395701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.395609    6430 generic.go:356] "Generic (PLEG): container finished" podID="7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" containerID="7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266" exitCode=0
Nov 03 08:39:25.395701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.395650    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99" event={"ID":"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2","Type":"ContainerDied","Data":"7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266"}
Nov 03 08:39:25.395701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.395653    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"
Nov 03 08:39:25.395967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.395725    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99" event={"ID":"7ed821a6-7a8b-44ec-b448-fa8c1702e6a2","Type":"ContainerDied","Data":"9b25109cb640e3e0c3fb670a65ae8c9bbb786cc668f417d1b9ca07fb0e45637d"}
Nov 03 08:39:25.395967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.395750    6430 scope.go:122] "RemoveContainer" containerID="7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266"
Nov 03 08:39:25.405390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.405375    6430 scope.go:122] "RemoveContainer" containerID="7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266"
Nov 03 08:39:25.405688 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:39:25.405667    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266\": container with ID starting with 7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266 not found: ID does not exist" containerID="7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266"
Nov 03 08:39:25.405745 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.405696    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266"} err="failed to get container status \"7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266\": rpc error: code = NotFound desc = could not find container \"7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266\": container with ID starting with 7913aba7b943b200bb9a514574b8df91bc45a5b75b492317240eb2fe21715266 not found: ID does not exist"
Nov 03 08:39:25.415201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.415177    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"]
Nov 03 08:39:25.418897 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:25.418873    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-890648-7b8456b499-ndc99"]
Nov 03 08:39:26.616554 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:26.616507    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" path="/var/lib/kubelet/pods/7ed821a6-7a8b-44ec-b448-fa8c1702e6a2/volumes"
Nov 03 08:39:28.374744 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:39:28.374706    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:39:28.849930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:28.849860    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" podUID="6aeed716-8861-4588-b7fe-a33508429567" containerName="payload-processing" containerID="cri-o://b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13" gracePeriod=25
Nov 03 08:39:28.868461 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:28.868413    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" podUID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" containerName="payload-pre-processing" containerID="cri-o://a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0" gracePeriod=25
Nov 03 08:39:30.015678 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:30.015642    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" podUID="6aeed716-8861-4588-b7fe-a33508429567" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:39:30.016125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:30.015648    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" podUID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:39:31.048986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:31.048952    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:31.185923 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:31.185880    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:31.322861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:31.322768    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"]
Nov 03 08:39:31.323074 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:31.323028    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy" containerID="cri-o://0c5052247d734162aae16c0169e72b945b0aa99f92d1733588512abca38f3b25" gracePeriod=30
Nov 03 08:39:31.470409 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:31.470375    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:33.567146 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.567052    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5" podUID="32bbf3ea-6531-4c5a-9509-05ec34940e7a" containerName="main" containerID="cri-o://64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2" gracePeriod=45
Nov 03 08:39:33.818650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.818597    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:39:33.926843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.926803    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-model-cache\") pod \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") "
Nov 03 08:39:33.927033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.926872    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-home\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-home\") pod \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") "
Nov 03 08:39:33.927033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.926905    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tls-certs\") pod \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") "
Nov 03 08:39:33.927033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.926944    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tmp-dir\") pod \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") "
Nov 03 08:39:33.927033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.926986    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32bbf3ea-6531-4c5a-9509-05ec34940e7a-kube-api-access-998tt\" (UniqueName: \"kubernetes.io/projected/32bbf3ea-6531-4c5a-9509-05ec34940e7a-kube-api-access-998tt\") pod \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") "
Nov 03 08:39:33.927314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.927045    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-dshm\") pod \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\" (UID: \"32bbf3ea-6531-4c5a-9509-05ec34940e7a\") "
Nov 03 08:39:33.927314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.927200    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-model-cache" pod "32bbf3ea-6531-4c5a-9509-05ec34940e7a" (UID: "32bbf3ea-6531-4c5a-9509-05ec34940e7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:33.927314 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.927213    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-home" pod "32bbf3ea-6531-4c5a-9509-05ec34940e7a" (UID: "32bbf3ea-6531-4c5a-9509-05ec34940e7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:33.927489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.927388    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:33.927489 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.927408    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:33.927615 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.927574    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tmp-dir" pod "32bbf3ea-6531-4c5a-9509-05ec34940e7a" (UID: "32bbf3ea-6531-4c5a-9509-05ec34940e7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:33.929359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.929333    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tls-certs" pod "32bbf3ea-6531-4c5a-9509-05ec34940e7a" (UID: "32bbf3ea-6531-4c5a-9509-05ec34940e7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:39:33.929682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.929667    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bbf3ea-6531-4c5a-9509-05ec34940e7a-kube-api-access-998tt" pod "32bbf3ea-6531-4c5a-9509-05ec34940e7a" (UID: "32bbf3ea-6531-4c5a-9509-05ec34940e7a"). InnerVolumeSpecName "kube-api-access-998tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:33.929757 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:33.929673    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-dshm" pod "32bbf3ea-6531-4c5a-9509-05ec34940e7a" (UID: "32bbf3ea-6531-4c5a-9509-05ec34940e7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:34.027957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.027927    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:34.028102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.027972    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:34.028102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.027983    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-998tt\" (UniqueName: \"kubernetes.io/projected/32bbf3ea-6531-4c5a-9509-05ec34940e7a-kube-api-access-998tt\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:34.028102 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.027993    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32bbf3ea-6531-4c5a-9509-05ec34940e7a-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:34.441738 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.441702    6430 generic.go:356] "Generic (PLEG): container finished" podID="32bbf3ea-6531-4c5a-9509-05ec34940e7a" containerID="64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2" exitCode=0
Nov 03 08:39:34.441946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.441744    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"
Nov 03 08:39:34.441946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.441762    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5" event={"ID":"32bbf3ea-6531-4c5a-9509-05ec34940e7a","Type":"ContainerDied","Data":"64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2"}
Nov 03 08:39:34.441946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.441816    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5" event={"ID":"32bbf3ea-6531-4c5a-9509-05ec34940e7a","Type":"ContainerDied","Data":"bf802261d67275c24cd88d0b8a85b3ceb60db041a212324c0535a22c5cb71ff2"}
Nov 03 08:39:34.441946 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.441838    6430 scope.go:122] "RemoveContainer" containerID="64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2"
Nov 03 08:39:34.452044 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.452029    6430 scope.go:122] "RemoveContainer" containerID="64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2"
Nov 03 08:39:34.452394 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:39:34.452374    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2\": container with ID starting with 64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2 not found: ID does not exist" containerID="64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2"
Nov 03 08:39:34.452458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.452401    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2"} err="failed to get container status \"64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2\": rpc error: code = NotFound desc = could not find container \"64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2\": container with ID starting with 64867dcacc2b36ff7852a18cff5da683fb93d9ed5cca0e3cfdf69b07ddbdd0a2 not found: ID does not exist"
Nov 03 08:39:34.464091 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.464046    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"]
Nov 03 08:39:34.469542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.469519    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-890648/simulator-single-main-890648-kserve-7b695c7599-pnvk5"]
Nov 03 08:39:34.615777 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:34.615744    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32bbf3ea-6531-4c5a-9509-05ec34940e7a" path="/var/lib/kubelet/pods/32bbf3ea-6531-4c5a-9509-05ec34940e7a/volumes"
Nov 03 08:39:35.629372 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:35.629339    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:39:35.629766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:35.629402    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:39:36.002221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.002184    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"]
Nov 03 08:39:36.002778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.002742    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" containerName="maas-api"
Nov 03 08:39:36.002971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.002954    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" containerName="maas-api"
Nov 03 08:39:36.003128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.003110    6430 cpu_manager.go:414] "containerMap: removing container" podUID="32bbf3ea-6531-4c5a-9509-05ec34940e7a" containerName="main"
Nov 03 08:39:36.003128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.003125    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bbf3ea-6531-4c5a-9509-05ec34940e7a" containerName="main"
Nov 03 08:39:36.003341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.003275    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed821a6-7a8b-44ec-b448-fa8c1702e6a2" containerName="maas-api"
Nov 03 08:39:36.003341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.003291    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bbf3ea-6531-4c5a-9509-05ec34940e7a" containerName="main"
Nov 03 08:39:36.006837 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.006816    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.010391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.010368    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.010502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.010449    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.010567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.010548    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vcxhj\""
Nov 03 08:39:36.010630 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.010596    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2aac1828-kserve-self-signed-certs\""
Nov 03 08:39:36.016795 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.016772    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"]
Nov 03 08:39:36.102816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.102771    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:36.145632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.145599    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-tmp-dir\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.145820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.145650    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-model-cache\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.145820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.145689    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-home\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.145914 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.145852    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-dshm\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.145977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.145929    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf921e-1488-4d67-8d34-533c4417098f-tls-certs\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.146045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.145982    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp82h\" (UniqueName: \"kubernetes.io/projected/10bf921e-1488-4d67-8d34-533c4417098f-kube-api-access-zp82h\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247019    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf921e-1488-4d67-8d34-533c4417098f-tls-certs\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247053 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247054    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp82h\" (UniqueName: \"kubernetes.io/projected/10bf921e-1488-4d67-8d34-533c4417098f-kube-api-access-zp82h\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247097    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-tmp-dir\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247117    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-model-cache\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247145    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-home\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247339 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247218    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-dshm\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247565 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247542    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-model-cache\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247628 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247583    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-home\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.247694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.247627    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-tmp-dir\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.249688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.249668    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-dshm\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.249861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.249844    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf921e-1488-4d67-8d34-533c4417098f-tls-certs\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.254567 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.254507    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp82h\" (UniqueName: \"kubernetes.io/projected/10bf921e-1488-4d67-8d34-533c4417098f-kube-api-access-zp82h\") pod \"e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") " pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.321149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.321121    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:36.455434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.455400    6430 generic.go:356] "Generic (PLEG): container finished" podID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerID="0c5052247d734162aae16c0169e72b945b0aa99f92d1733588512abca38f3b25" exitCode=0
Nov 03 08:39:36.455597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.455477    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" event={"ID":"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2","Type":"ContainerDied","Data":"0c5052247d734162aae16c0169e72b945b0aa99f92d1733588512abca38f3b25"}
Nov 03 08:39:36.455597 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.455551    6430 scope.go:122] "RemoveContainer" containerID="c10ca23e794186311e4372c102f388f1736ba35d82cb20354635344e3138920f"
Nov 03 08:39:36.559557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.559535    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:39:36.603434 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.603381    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"]
Nov 03 08:39:36.607195 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:39:36.607169    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10bf921e_1488_4d67_8d34_533c4417098f.slice/crio-9afcf4a7ca0c823357e55e90f5e4c8e6b852e61f9b377170af5ab12a4c8e4eaf WatchSource:0}: Error finding container 9afcf4a7ca0c823357e55e90f5e4c8e6b852e61f9b377170af5ab12a4c8e4eaf: Status 404 returned error can't find the container with id 9afcf4a7ca0c823357e55e90f5e4c8e6b852e61f9b377170af5ab12a4c8e4eaf
Nov 03 08:39:36.650931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.650910    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-token\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-token\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.650961    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-data\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.650980    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-credential-socket\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651023    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-certs\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651094    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-podinfo\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651130    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-socket\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651274    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istiod-ca-cert\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651285    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-data" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:36.651325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651302    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-envoy\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651360    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-kube-api-access-n82f6\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-kube-api-access-n82f6\") pod \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\" (UID: \"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2\") "
Nov 03 08:39:36.651661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651589    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.651821 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651799    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-certs" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:36.652161 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.651890    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-socket" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:36.652250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.652185    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istiod-ca-cert" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:39:36.652295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.652256    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-credential-socket" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:36.653570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.653537    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-token" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:36.654213 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.654193    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-kube-api-access-n82f6" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "kube-api-access-n82f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:36.654374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.654340    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-podinfo" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:39:36.654437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.654354    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-envoy" pod "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" (UID: "3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:39:36.752775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752745    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.752775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752772    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.752775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752783    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.753027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752793    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.753027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752802    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.753027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752812    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.753027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752820    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82f6\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-kube-api-access-n82f6\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:36.753027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:36.752829    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:37.461392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.461339    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk" event={"ID":"10bf921e-1488-4d67-8d34-533c4417098f","Type":"ContainerStarted","Data":"ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7"}
Nov 03 08:39:37.461392 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.461395    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk" event={"ID":"10bf921e-1488-4d67-8d34-533c4417098f","Type":"ContainerStarted","Data":"9afcf4a7ca0c823357e55e90f5e4c8e6b852e61f9b377170af5ab12a4c8e4eaf"}
Nov 03 08:39:37.462715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.462693    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"
Nov 03 08:39:37.462816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.462695    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs" event={"ID":"3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2","Type":"ContainerDied","Data":"0eef07118288581563f0214a549fc1542c67384ac9e0e2f8555e104b4d3f2d5a"}
Nov 03 08:39:37.462867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.462826    6430 scope.go:122] "RemoveContainer" containerID="0c5052247d734162aae16c0169e72b945b0aa99f92d1733588512abca38f3b25"
Nov 03 08:39:37.480366 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.480319    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk" podStartSLOduration=2.480306226 podStartE2EDuration="2.480306226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:37.478772539 +0000 UTC m=+2726.981608054" watchObservedRunningTime="2026-11-03 08:39:37.480306226 +0000 UTC m=+2726.983141739"
Nov 03 08:39:37.498173 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.498148    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"]
Nov 03 08:39:37.502285 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:37.502262    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-890648-openshift-default-db9cbd785-6wsvs"]
Nov 03 08:39:38.321521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:38.321475    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:38.617894 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:38.617797    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" path="/var/lib/kubelet/pods/3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2/volumes"
Nov 03 08:39:40.029086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:40.028142    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" podUID="6aeed716-8861-4588-b7fe-a33508429567" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:39:40.029086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:40.028610    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" podUID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:39:46.322374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:46.322336    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:46.334992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:46.334969    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:46.525032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:46.525006    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:39:49.110304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.110244    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:39:49.137861 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.137838    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:39:49.178717 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.178689    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeed716-8861-4588-b7fe-a33508429567-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aeed716-8861-4588-b7fe-a33508429567-plugins-config-volume\") pod \"6aeed716-8861-4588-b7fe-a33508429567\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") "
Nov 03 08:39:49.178839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.178739    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aeed716-8861-4588-b7fe-a33508429567-kube-api-access-fnqm6\" (UniqueName: \"kubernetes.io/projected/6aeed716-8861-4588-b7fe-a33508429567-kube-api-access-fnqm6\") pod \"6aeed716-8861-4588-b7fe-a33508429567\" (UID: \"6aeed716-8861-4588-b7fe-a33508429567\") "
Nov 03 08:39:49.179202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.179171    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeed716-8861-4588-b7fe-a33508429567-plugins-config-volume" pod "6aeed716-8861-4588-b7fe-a33508429567" (UID: "6aeed716-8861-4588-b7fe-a33508429567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:39:49.180957 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.180936    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aeed716-8861-4588-b7fe-a33508429567-kube-api-access-fnqm6" pod "6aeed716-8861-4588-b7fe-a33508429567" (UID: "6aeed716-8861-4588-b7fe-a33508429567"). InnerVolumeSpecName "kube-api-access-fnqm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:49.279705 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.279677    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-kube-api-access-kkpsn\" (UniqueName: \"kubernetes.io/projected/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-kube-api-access-kkpsn\") pod \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") "
Nov 03 08:39:49.279856 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.279716    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-plugins-config-volume\") pod \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\" (UID: \"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a\") "
Nov 03 08:39:49.280024 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.280005    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aeed716-8861-4588-b7fe-a33508429567-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:49.280088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.280032    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnqm6\" (UniqueName: \"kubernetes.io/projected/6aeed716-8861-4588-b7fe-a33508429567-kube-api-access-fnqm6\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:49.280189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.280173    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-plugins-config-volume" pod "a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" (UID: "a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:39:49.281832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.281812    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-kube-api-access-kkpsn" pod "a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" (UID: "a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a"). InnerVolumeSpecName "kube-api-access-kkpsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:49.381526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.381466    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkpsn\" (UniqueName: \"kubernetes.io/projected/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-kube-api-access-kkpsn\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:49.381526 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.381497    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:49.530477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.530446    6430 generic.go:356] "Generic (PLEG): container finished" podID="6aeed716-8861-4588-b7fe-a33508429567" containerID="b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13" exitCode=0
Nov 03 08:39:49.530599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.530493    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"
Nov 03 08:39:49.530599 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.530536    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" event={"ID":"6aeed716-8861-4588-b7fe-a33508429567","Type":"ContainerDied","Data":"b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13"}
Nov 03 08:39:49.530694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.530599    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq" event={"ID":"6aeed716-8861-4588-b7fe-a33508429567","Type":"ContainerDied","Data":"c36bb7c5c638146b36d9d00cdf1cb4a28193b404199ff7fc7a28c89dc310cc3c"}
Nov 03 08:39:49.530694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.530632    6430 scope.go:122] "RemoveContainer" containerID="b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13"
Nov 03 08:39:49.531964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.531939    6430 generic.go:356] "Generic (PLEG): container finished" podID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" containerID="a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0" exitCode=0
Nov 03 08:39:49.532085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.531989    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"
Nov 03 08:39:49.532085 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.532028    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" event={"ID":"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a","Type":"ContainerDied","Data":"a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0"}
Nov 03 08:39:49.532179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.532088    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67" event={"ID":"a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a","Type":"ContainerDied","Data":"62036cf56033f79fa908a3f7a6c7f570abb7a187473313ecb117df7a38f1ff71"}
Nov 03 08:39:49.539328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.539303    6430 scope.go:122] "RemoveContainer" containerID="b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13"
Nov 03 08:39:49.539622 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:39:49.539600    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13\": container with ID starting with b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13 not found: ID does not exist" containerID="b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13"
Nov 03 08:39:49.539713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.539629    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13"} err="failed to get container status \"b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13\": rpc error: code = NotFound desc = could not find container \"b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13\": container with ID starting with b6a90ce296d4cf3b94a7a1a9aece89e6433d3eb10b334d66914e2c93562e9d13 not found: ID does not exist"
Nov 03 08:39:49.539713 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.539645    6430 scope.go:122] "RemoveContainer" containerID="a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0"
Nov 03 08:39:49.547701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.547685    6430 scope.go:122] "RemoveContainer" containerID="a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0"
Nov 03 08:39:49.547963 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:39:49.547944    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0\": container with ID starting with a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0 not found: ID does not exist" containerID="a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0"
Nov 03 08:39:49.548027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.547974    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0"} err="failed to get container status \"a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0\": rpc error: code = NotFound desc = could not find container \"a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0\": container with ID starting with a7d0260114a6f85d15ba51eae2da3a7fc24135df6e690c20325a7ed05c1917e0 not found: ID does not exist"
Nov 03 08:39:49.554232 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.554212    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"]
Nov 03 08:39:49.559047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.559028    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-890648-8545496m67"]
Nov 03 08:39:49.567477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.567463    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"]
Nov 03 08:39:49.571256 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:49.571234    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-890648-5657fbf7fpk6zq"]
Nov 03 08:39:50.614754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:50.614719    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aeed716-8861-4588-b7fe-a33508429567" path="/var/lib/kubelet/pods/6aeed716-8861-4588-b7fe-a33508429567/volumes"
Nov 03 08:39:50.615171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:50.615098    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" path="/var/lib/kubelet/pods/a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a/volumes"
Nov 03 08:39:51.210388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.210352    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"]
Nov 03 08:39:51.210910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.210888    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy"
Nov 03 08:39:51.211016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.210913    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy"
Nov 03 08:39:51.211016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.210950    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy"
Nov 03 08:39:51.211016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.210960    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy"
Nov 03 08:39:51.211016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.210983    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6aeed716-8861-4588-b7fe-a33508429567" containerName="payload-processing"
Nov 03 08:39:51.211016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.210990    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aeed716-8861-4588-b7fe-a33508429567" containerName="payload-processing"
Nov 03 08:39:51.211016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.211008    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" containerName="payload-pre-processing"
Nov 03 08:39:51.211016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.211015    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" containerName="payload-pre-processing"
Nov 03 08:39:51.211298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.211122    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy"
Nov 03 08:39:51.211298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.211138    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c14b18c-6836-4ed5-b22d-ab4fa98bf9e2" containerName="istio-proxy"
Nov 03 08:39:51.211298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.211149    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aeed716-8861-4588-b7fe-a33508429567" containerName="payload-processing"
Nov 03 08:39:51.211298 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.211161    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ec5be7-cd3b-44bf-b1d2-731b0ddced4a" containerName="payload-pre-processing"
Nov 03 08:39:51.216290 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.216264    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.226432 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.226401    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2aac1828-kserve-self-signed-certs\""
Nov 03 08:39:51.227332 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.227309    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"]
Nov 03 08:39:51.298940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.298900    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bcjc\" (UniqueName: \"kubernetes.io/projected/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-kube-api-access-7bcjc\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.299177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.298950    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-model-cache\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.299177 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.299138    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tmp-dir\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.299310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.299186    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-dshm\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.299310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.299273    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tls-certs\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.299310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.299304    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-home\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.311292 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.311264    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:51.400655 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.400622    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bcjc\" (UniqueName: \"kubernetes.io/projected/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-kube-api-access-7bcjc\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.400906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.400668    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-model-cache\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.400906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.400703    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tmp-dir\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.400906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.400736    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-dshm\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.400906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.400768    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tls-certs\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.400906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.400797    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-home\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.401328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.401167    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-model-cache\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.401328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.401272    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tmp-dir\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.401328 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.401297    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-home\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.403574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.403547    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tls-certs\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.403574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.403564    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-dshm\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.407883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.407859    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bcjc\" (UniqueName: \"kubernetes.io/projected/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-kube-api-access-7bcjc\") pod \"e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") " pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.529938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.529867    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:51.810463 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:51.810381    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"]
Nov 03 08:39:51.814191 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:39:51.814154    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14ebe48c_b207_4bcb_b2c4_0aaa36deae7d.slice/crio-6732038346de4a2fa743439732175bf1e4e28eceba4638bbd5fb386f963a97e5 WatchSource:0}: Error finding container 6732038346de4a2fa743439732175bf1e4e28eceba4638bbd5fb386f963a97e5: Status 404 returned error can't find the container with id 6732038346de4a2fa743439732175bf1e4e28eceba4638bbd5fb386f963a97e5
Nov 03 08:39:52.551483 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:52.551436    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz" event={"ID":"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d","Type":"ContainerStarted","Data":"05bc28da6d1e1e8c28075a0a8ce400740446c5303007e8df78517fe41fad98c9"}
Nov 03 08:39:52.551688 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:52.551496    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz" event={"ID":"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d","Type":"ContainerStarted","Data":"6732038346de4a2fa743439732175bf1e4e28eceba4638bbd5fb386f963a97e5"}
Nov 03 08:39:52.569486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:52.569440    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz" podStartSLOduration=1.569426196 podStartE2EDuration="1.569426196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:52.567929412 +0000 UTC m=+2742.070764925" watchObservedRunningTime="2026-11-03 08:39:52.569426196 +0000 UTC m=+2742.072261710"
Nov 03 08:39:53.530809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:53.530778    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:39:55.417087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:55.416854    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:55.915087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:39:55.911492    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:58.374845 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:39:58.374806    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:40:01.472167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:01.472136    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:40:01.530431 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:01.530404    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:40:01.542814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:01.542795    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:40:01.625331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:01.625302    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:40:12.895274 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:12.895236    6430 scope.go:122] "RemoveContainer" containerID="f41b0103e5b84db445e99bfc80b8a6dbccd5692b1b1f472cd5fe039864fcb7c0"
Nov 03 08:40:12.903479 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:12.903455    6430 scope.go:122] "RemoveContainer" containerID="d95cc342f419ecdb8ec781e2aa45d73ab43b7c9b4f15b536ddd14ed9651d9226"
Nov 03 08:40:12.910797 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:12.910780    6430 scope.go:122] "RemoveContainer" containerID="06d7e3342b27657d6286db9eb4fbc151cd3fb052530f2349994972e10746595e"
Nov 03 08:40:12.918150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:12.918133    6430 scope.go:122] "RemoveContainer" containerID="3d38d263c485a256b31595cdf15dff98ec29fc39bd826ef4b88322e09cdcc344"
Nov 03 08:40:15.128422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:15.128375    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:40:19.248385 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:19.248340    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"]
Nov 03 08:40:19.330608 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:19.330571    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:40:20.627736 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:20.627699    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"]
Nov 03 08:40:20.678286 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:20.678251    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:40:22.748097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:22.747430    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:40:28.374023 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:40:28.373987    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:40:34.276460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.276390    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk" podUID="10bf921e-1488-4d67-8d34-533c4417098f" containerName="main" containerID="cri-o://ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7" gracePeriod=45
Nov 03 08:40:34.682108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.682075    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:40:34.757464 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.757430    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-tmp-dir\") pod \"10bf921e-1488-4d67-8d34-533c4417098f\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") "
Nov 03 08:40:34.757800 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.757777    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-model-cache\") pod \"10bf921e-1488-4d67-8d34-533c4417098f\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") "
Nov 03 08:40:34.758021 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.757990    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bf921e-1488-4d67-8d34-533c4417098f-tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf921e-1488-4d67-8d34-533c4417098f-tls-certs\") pod \"10bf921e-1488-4d67-8d34-533c4417098f\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") "
Nov 03 08:40:34.762271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.762240    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-home\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-home\") pod \"10bf921e-1488-4d67-8d34-533c4417098f\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") "
Nov 03 08:40:34.768683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.768661    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-dshm\") pod \"10bf921e-1488-4d67-8d34-533c4417098f\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") "
Nov 03 08:40:34.777096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.768908    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bf921e-1488-4d67-8d34-533c4417098f-kube-api-access-zp82h\" (UniqueName: \"kubernetes.io/projected/10bf921e-1488-4d67-8d34-533c4417098f-kube-api-access-zp82h\") pod \"10bf921e-1488-4d67-8d34-533c4417098f\" (UID: \"10bf921e-1488-4d67-8d34-533c4417098f\") "
Nov 03 08:40:34.778394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.767147    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-home" pod "10bf921e-1488-4d67-8d34-533c4417098f" (UID: "10bf921e-1488-4d67-8d34-533c4417098f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:34.778394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.767426    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-tmp-dir" pod "10bf921e-1488-4d67-8d34-533c4417098f" (UID: "10bf921e-1488-4d67-8d34-533c4417098f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:34.778394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.768145    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-model-cache" pod "10bf921e-1488-4d67-8d34-533c4417098f" (UID: "10bf921e-1488-4d67-8d34-533c4417098f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:34.780683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.780645    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-dshm" pod "10bf921e-1488-4d67-8d34-533c4417098f" (UID: "10bf921e-1488-4d67-8d34-533c4417098f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:34.781120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.781092    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bf921e-1488-4d67-8d34-533c4417098f-tls-certs" pod "10bf921e-1488-4d67-8d34-533c4417098f" (UID: "10bf921e-1488-4d67-8d34-533c4417098f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:34.782382 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.782356    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bf921e-1488-4d67-8d34-533c4417098f-kube-api-access-zp82h" pod "10bf921e-1488-4d67-8d34-533c4417098f" (UID: "10bf921e-1488-4d67-8d34-533c4417098f"). InnerVolumeSpecName "kube-api-access-zp82h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:40:34.868144 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.868012    6430 generic.go:356] "Generic (PLEG): container finished" podID="10bf921e-1488-4d67-8d34-533c4417098f" containerID="ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7" exitCode=0
Nov 03 08:40:34.868325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.868155    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk" event={"ID":"10bf921e-1488-4d67-8d34-533c4417098f","Type":"ContainerDied","Data":"ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7"}
Nov 03 08:40:34.868325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.868195    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk" event={"ID":"10bf921e-1488-4d67-8d34-533c4417098f","Type":"ContainerDied","Data":"9afcf4a7ca0c823357e55e90f5e4c8e6b852e61f9b377170af5ab12a4c8e4eaf"}
Nov 03 08:40:34.868325 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.868217    6430 scope.go:122] "RemoveContainer" containerID="ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7"
Nov 03 08:40:34.868521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.868374    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"
Nov 03 08:40:34.878419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.878310    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp82h\" (UniqueName: \"kubernetes.io/projected/10bf921e-1488-4d67-8d34-533c4417098f-kube-api-access-zp82h\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:34.878419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.878338    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:34.878419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.878354    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:34.878419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.878369    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf921e-1488-4d67-8d34-533c4417098f-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:34.878419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.878383    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:34.878419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.878396    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf921e-1488-4d67-8d34-533c4417098f-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:34.881769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.881716    6430 scope.go:122] "RemoveContainer" containerID="ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7"
Nov 03 08:40:34.882018 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:40:34.881997    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7\": container with ID starting with ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7 not found: ID does not exist" containerID="ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7"
Nov 03 08:40:34.882141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.882029    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7"} err="failed to get container status \"ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7\": rpc error: code = NotFound desc = could not find container \"ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7\": container with ID starting with ba6c679c58b1017529b5f32a7b3acf470a63098abded6b1a19f7a3d45e9782c7 not found: ID does not exist"
Nov 03 08:40:34.894559 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.894528    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"]
Nov 03 08:40:34.921416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:34.921364    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk"]
Nov 03 08:40:35.525704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:35.525662    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-2aac1828-kserve-6f978d8c4f-rjfvk" podUID="10bf921e-1488-4d67-8d34-533c4417098f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:40:35.649299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:35.649259    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz" podUID="14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" containerName="main" containerID="cri-o://05bc28da6d1e1e8c28075a0a8ce400740446c5303007e8df78517fe41fad98c9" gracePeriod=45
Nov 03 08:40:35.873754 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:35.873717    6430 generic.go:356] "Generic (PLEG): container finished" podID="14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" containerID="05bc28da6d1e1e8c28075a0a8ce400740446c5303007e8df78517fe41fad98c9" exitCode=0
Nov 03 08:40:35.873922 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:35.873792    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz" event={"ID":"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d","Type":"ContainerDied","Data":"05bc28da6d1e1e8c28075a0a8ce400740446c5303007e8df78517fe41fad98c9"}
Nov 03 08:40:35.902815 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:35.902790    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:40:36.088804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.088712    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tls-certs\" (UniqueName: \"kubernetes.io/secret/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tls-certs\") pod \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") "
Nov 03 08:40:36.088804 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.088774    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tmp-dir\") pod \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") "
Nov 03 08:40:36.089033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.088946    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-kube-api-access-7bcjc\" (UniqueName: \"kubernetes.io/projected/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-kube-api-access-7bcjc\") pod \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") "
Nov 03 08:40:36.089126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089084    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-model-cache\") pod \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") "
Nov 03 08:40:36.089126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089087    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tmp-dir" pod "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" (UID: "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:36.089260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089130    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-home\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-home\") pod \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") "
Nov 03 08:40:36.089260 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089187    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-dshm\") pod \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\" (UID: \"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d\") "
Nov 03 08:40:36.089397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089374    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-model-cache" pod "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" (UID: "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:36.089541 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089521    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tmp-dir\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:36.089612 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089546    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-model-cache\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:36.089677 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.089655    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-home" pod "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" (UID: "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:36.092027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.091945    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tls-certs" pod "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" (UID: "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:36.092027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.091955    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-kube-api-access-7bcjc" pod "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" (UID: "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d"). InnerVolumeSpecName "kube-api-access-7bcjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:40:36.093472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.093437    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-dshm" pod "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" (UID: "14ebe48c-b207-4bcb-b2c4-0aaa36deae7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:36.190263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.190218    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-tls-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:36.190263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.190265    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bcjc\" (UniqueName: \"kubernetes.io/projected/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-kube-api-access-7bcjc\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:36.190263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.190283    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-home\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:36.190643 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.190300    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d-dshm\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:36.616381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.616342    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bf921e-1488-4d67-8d34-533c4417098f" path="/var/lib/kubelet/pods/10bf921e-1488-4d67-8d34-533c4417098f/volumes"
Nov 03 08:40:36.882727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.882635    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz" event={"ID":"14ebe48c-b207-4bcb-b2c4-0aaa36deae7d","Type":"ContainerDied","Data":"6732038346de4a2fa743439732175bf1e4e28eceba4638bbd5fb386f963a97e5"}
Nov 03 08:40:36.882727 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.882704    6430 scope.go:122] "RemoveContainer" containerID="05bc28da6d1e1e8c28075a0a8ce400740446c5303007e8df78517fe41fad98c9"
Nov 03 08:40:36.882980 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.882649    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"
Nov 03 08:40:36.899206 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.899175    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"]
Nov 03 08:40:36.912137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:36.912104    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2aac1828-kserve-799db8c65-z5vhz"]
Nov 03 08:40:38.620125 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:38.620086    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" path="/var/lib/kubelet/pods/14ebe48c-b207-4bcb-b2c4-0aaa36deae7d/volumes"
Nov 03 08:40:42.329764 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.329726    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"]
Nov 03 08:40:42.330203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.330153    6430 cpu_manager.go:414] "containerMap: removing container" podUID="10bf921e-1488-4d67-8d34-533c4417098f" containerName="main"
Nov 03 08:40:42.330203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.330168    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bf921e-1488-4d67-8d34-533c4417098f" containerName="main"
Nov 03 08:40:42.330203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.330180    6430 cpu_manager.go:414] "containerMap: removing container" podUID="14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" containerName="main"
Nov 03 08:40:42.330203 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.330187    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" containerName="main"
Nov 03 08:40:42.330344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.330268    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bf921e-1488-4d67-8d34-533c4417098f" containerName="main"
Nov 03 08:40:42.330344 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.330275    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ebe48c-b207-4bcb-b2c4-0aaa36deae7d" containerName="main"
Nov 03 08:40:42.334721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.334702    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.339798 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.339774    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:40:42.340430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.340407    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:40:42.342009 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.341783    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"]
Nov 03 08:40:42.386968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.386930    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknjl\" (UniqueName: \"kubernetes.io/projected/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-kube-api-access-hknjl\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.387209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.387000    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.387209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.387119    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.488264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.488226    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.488445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.488277    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.488445 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.488332    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknjl\" (UniqueName: \"kubernetes.io/projected/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-kube-api-access-hknjl\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.488873 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.488850    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.490891 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.490871    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.495480 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.495455    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknjl\" (UniqueName: \"kubernetes.io/projected/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-kube-api-access-hknjl\") pod \"maas-api-revoke-sub-a1bd9efb-j5wxd\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.649276 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.649201    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:42.921120 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:42.921024    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"]
Nov 03 08:40:42.924956 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:40:42.924927    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8326ebd_5944_4248_9ca3_cd5c5ba470ac.slice/crio-6ee30c401a4849334eda5003b5f5ea6a564fae50d7f27e4f7f4a2667f9857f58 WatchSource:0}: Error finding container 6ee30c401a4849334eda5003b5f5ea6a564fae50d7f27e4f7f4a2667f9857f58: Status 404 returned error can't find the container with id 6ee30c401a4849334eda5003b5f5ea6a564fae50d7f27e4f7f4a2667f9857f58
Nov 03 08:40:43.916842 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:43.916801    6430 generic.go:356] "Generic (PLEG): container finished" podID="e8326ebd-5944-4248-9ca3-cd5c5ba470ac" containerID="fb1f9c640fe465c65b7cc483853570d756e9d908294d82d4624b6468c7cd6788" exitCode=0
Nov 03 08:40:43.917363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:43.916885    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd" event={"ID":"e8326ebd-5944-4248-9ca3-cd5c5ba470ac","Type":"ContainerDied","Data":"fb1f9c640fe465c65b7cc483853570d756e9d908294d82d4624b6468c7cd6788"}
Nov 03 08:40:43.917363 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:43.916940    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd" event={"ID":"e8326ebd-5944-4248-9ca3-cd5c5ba470ac","Type":"ContainerStarted","Data":"6ee30c401a4849334eda5003b5f5ea6a564fae50d7f27e4f7f4a2667f9857f58"}
Nov 03 08:40:45.047904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.047882    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:45.107417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.107388    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-kube-api-access-hknjl\" (UniqueName: \"kubernetes.io/projected/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-kube-api-access-hknjl\") pod \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") "
Nov 03 08:40:45.107570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.107428    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-xks-ca\") pod \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") "
Nov 03 08:40:45.107570 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.107486    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-openshift-ca\") pod \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\" (UID: \"e8326ebd-5944-4248-9ca3-cd5c5ba470ac\") "
Nov 03 08:40:45.108006 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.107978    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-openshift-ca" pod "e8326ebd-5944-4248-9ca3-cd5c5ba470ac" (UID: "e8326ebd-5944-4248-9ca3-cd5c5ba470ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:40:45.109809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.109780    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-xks-ca" pod "e8326ebd-5944-4248-9ca3-cd5c5ba470ac" (UID: "e8326ebd-5944-4248-9ca3-cd5c5ba470ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:45.109809 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.109783    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-kube-api-access-hknjl" pod "e8326ebd-5944-4248-9ca3-cd5c5ba470ac" (UID: "e8326ebd-5944-4248-9ca3-cd5c5ba470ac"). InnerVolumeSpecName "kube-api-access-hknjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:40:45.208780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.208754    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknjl\" (UniqueName: \"kubernetes.io/projected/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-kube-api-access-hknjl\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:45.208780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.208779    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-xks-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:45.208916 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.208790    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8326ebd-5944-4248-9ca3-cd5c5ba470ac-maas-api-openshift-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:45.927395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.927362    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"
Nov 03 08:40:45.927571 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.927362    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd" event={"ID":"e8326ebd-5944-4248-9ca3-cd5c5ba470ac","Type":"ContainerDied","Data":"6ee30c401a4849334eda5003b5f5ea6a564fae50d7f27e4f7f4a2667f9857f58"}
Nov 03 08:40:45.927571 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:45.927487    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ee30c401a4849334eda5003b5f5ea6a564fae50d7f27e4f7f4a2667f9857f58"
Nov 03 08:40:54.443043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.442945    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"]
Nov 03 08:40:54.443462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.443412    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e8326ebd-5944-4248-9ca3-cd5c5ba470ac" containerName="revoke-keys"
Nov 03 08:40:54.443462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.443427    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8326ebd-5944-4248-9ca3-cd5c5ba470ac" containerName="revoke-keys"
Nov 03 08:40:54.443550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.443492    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8326ebd-5944-4248-9ca3-cd5c5ba470ac" containerName="revoke-keys"
Nov 03 08:40:54.446890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.446868    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.451406 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.451381    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 08:40:54.452390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.451702    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7nz66\""
Nov 03 08:40:54.452560 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.451778    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 08:40:54.457178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.457154    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"]
Nov 03 08:40:54.460320 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.460295    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"]
Nov 03 08:40:54.464108 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.464089    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.479728 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.479699    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"]
Nov 03 08:40:54.491469 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491435    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-maas-api-tls\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.491636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491513    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9508d41-956b-49ec-8306-a1ea83904f0b-openshift-service-ca\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.491636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491540    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978pm\" (UniqueName: \"kubernetes.io/projected/f9508d41-956b-49ec-8306-a1ea83904f0b-kube-api-access-978pm\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.491636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491601    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9508d41-956b-49ec-8306-a1ea83904f0b-metrics-certs\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.491636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491629    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-metrics-certs\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.491870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491657    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef49c105-774b-4caa-82e0-d00f2f1c38b1-openshift-service-ca\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.491870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491714    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9508d41-956b-49ec-8306-a1ea83904f0b-maas-api-tls\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.491870 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.491778    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5jj\" (UniqueName: \"kubernetes.io/projected/ef49c105-774b-4caa-82e0-d00f2f1c38b1-kube-api-access-nd5jj\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.592938 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.592905    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5jj\" (UniqueName: \"kubernetes.io/projected/ef49c105-774b-4caa-82e0-d00f2f1c38b1-kube-api-access-nd5jj\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.593160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.592969    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-maas-api-tls\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.593160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593025    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9508d41-956b-49ec-8306-a1ea83904f0b-openshift-service-ca\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.593160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593049    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978pm\" (UniqueName: \"kubernetes.io/projected/f9508d41-956b-49ec-8306-a1ea83904f0b-kube-api-access-978pm\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.593160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593108    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9508d41-956b-49ec-8306-a1ea83904f0b-metrics-certs\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.593160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593131    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-metrics-certs\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.593160 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593155    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef49c105-774b-4caa-82e0-d00f2f1c38b1-openshift-service-ca\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.593515 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593324    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9508d41-956b-49ec-8306-a1ea83904f0b-maas-api-tls\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.593935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593905    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef49c105-774b-4caa-82e0-d00f2f1c38b1-openshift-service-ca\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.594114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.593886    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9508d41-956b-49ec-8306-a1ea83904f0b-openshift-service-ca\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.595557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.595527    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-maas-api-tls\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.595725 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.595701    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9508d41-956b-49ec-8306-a1ea83904f0b-metrics-certs\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.595768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.595751    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-metrics-certs\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.596010 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.595989    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9508d41-956b-49ec-8306-a1ea83904f0b-maas-api-tls\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.600689 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.600661    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5jj\" (UniqueName: \"kubernetes.io/projected/ef49c105-774b-4caa-82e0-d00f2f1c38b1-kube-api-access-nd5jj\") pod \"maas-api-7765546dfd-vnc7w\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.601534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.601511    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-978pm\" (UniqueName: \"kubernetes.io/projected/f9508d41-956b-49ec-8306-a1ea83904f0b-kube-api-access-978pm\") pod \"maas-api-6f6d4c9f4b-wkd89\" (UID: \"f9508d41-956b-49ec-8306-a1ea83904f0b\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:54.766784 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.766694    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:54.784310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:54.784284    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:55.051528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.051442    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"]
Nov 03 08:40:55.055035 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:40:55.055008    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef49c105_774b_4caa_82e0_d00f2f1c38b1.slice/crio-bc2146c3dee4cd595e6e38d993719a539136bf949004e4b495d08d8c95c5cbae WatchSource:0}: Error finding container bc2146c3dee4cd595e6e38d993719a539136bf949004e4b495d08d8c95c5cbae: Status 404 returned error can't find the container with id bc2146c3dee4cd595e6e38d993719a539136bf949004e4b495d08d8c95c5cbae
Nov 03 08:40:55.068005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.067979    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"]
Nov 03 08:40:55.071397 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:40:55.071368    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9508d41_956b_49ec_8306_a1ea83904f0b.slice/crio-3e43f8b5b83c73c089e236b3d3c8b35aeec511795a456047fddd166c2abd54e0 WatchSource:0}: Error finding container 3e43f8b5b83c73c089e236b3d3c8b35aeec511795a456047fddd166c2abd54e0: Status 404 returned error can't find the container with id 3e43f8b5b83c73c089e236b3d3c8b35aeec511795a456047fddd166c2abd54e0
Nov 03 08:40:55.971557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.971524    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89" event={"ID":"f9508d41-956b-49ec-8306-a1ea83904f0b","Type":"ContainerStarted","Data":"71a2364e3ddad31071d0613695a2357d5550a57cbf2b75d00af6da17b48654c7"}
Nov 03 08:40:55.972017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.971577    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89" event={"ID":"f9508d41-956b-49ec-8306-a1ea83904f0b","Type":"ContainerStarted","Data":"3e43f8b5b83c73c089e236b3d3c8b35aeec511795a456047fddd166c2abd54e0"}
Nov 03 08:40:55.972017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.971629    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:40:55.972944 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.972916    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w" event={"ID":"ef49c105-774b-4caa-82e0-d00f2f1c38b1","Type":"ContainerStarted","Data":"ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734"}
Nov 03 08:40:55.973107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.972951    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w" event={"ID":"ef49c105-774b-4caa-82e0-d00f2f1c38b1","Type":"ContainerStarted","Data":"bc2146c3dee4cd595e6e38d993719a539136bf949004e4b495d08d8c95c5cbae"}
Nov 03 08:40:55.973107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.973053    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:40:55.986779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:55.986730    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89" podStartSLOduration=1.504632497 podStartE2EDuration="1.98671978s" totalImagesPullingTime="482.087283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:55.985746604 +0000 UTC m=+2805.488582117" watchObservedRunningTime="2026-11-03 08:40:55.98671978 +0000 UTC m=+2805.489555319"
Nov 03 08:40:56.001875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:40:56.001835    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w" podStartSLOduration=1.46729987 podStartE2EDuration="2.00182384s" totalImagesPullingTime="534.52397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:56.000832038 +0000 UTC m=+2805.503667572" watchObservedRunningTime="2026-11-03 08:40:56.00182384 +0000 UTC m=+2805.504659348"
Nov 03 08:40:58.375136 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:40:58.375098    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:41:01.152653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:01.152619    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:01.561956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:01.561923    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:01.685187 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:01.685151    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:01.983895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:01.983867    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-wkd89"
Nov 03 08:41:01.984087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:01.983944    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:41:02.047767 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.047730    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"]
Nov 03 08:41:02.047971 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.047946    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w" podUID="ef49c105-774b-4caa-82e0-d00f2f1c38b1" containerName="maas-api" containerID="cri-o://ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734" gracePeriod=30
Nov 03 08:41:02.295585 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.295561    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:41:02.366814 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.366788    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef49c105-774b-4caa-82e0-d00f2f1c38b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef49c105-774b-4caa-82e0-d00f2f1c38b1-openshift-service-ca\") pod \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") "
Nov 03 08:41:02.366997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.366844    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-metrics-certs\") pod \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") "
Nov 03 08:41:02.366997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.366867    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef49c105-774b-4caa-82e0-d00f2f1c38b1-kube-api-access-nd5jj\" (UniqueName: \"kubernetes.io/projected/ef49c105-774b-4caa-82e0-d00f2f1c38b1-kube-api-access-nd5jj\") pod \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") "
Nov 03 08:41:02.366997 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.366940    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-maas-api-tls\") pod \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\" (UID: \"ef49c105-774b-4caa-82e0-d00f2f1c38b1\") "
Nov 03 08:41:02.367250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.367229    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef49c105-774b-4caa-82e0-d00f2f1c38b1-openshift-service-ca" pod "ef49c105-774b-4caa-82e0-d00f2f1c38b1" (UID: "ef49c105-774b-4caa-82e0-d00f2f1c38b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:02.369143 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.369123    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-metrics-certs" pod "ef49c105-774b-4caa-82e0-d00f2f1c38b1" (UID: "ef49c105-774b-4caa-82e0-d00f2f1c38b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:02.369241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.369145    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-maas-api-tls" pod "ef49c105-774b-4caa-82e0-d00f2f1c38b1" (UID: "ef49c105-774b-4caa-82e0-d00f2f1c38b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:02.369241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.369212    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef49c105-774b-4caa-82e0-d00f2f1c38b1-kube-api-access-nd5jj" pod "ef49c105-774b-4caa-82e0-d00f2f1c38b1" (UID: "ef49c105-774b-4caa-82e0-d00f2f1c38b1"). InnerVolumeSpecName "kube-api-access-nd5jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:02.468291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.468262    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:02.468291 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.468287    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd5jj\" (UniqueName: \"kubernetes.io/projected/ef49c105-774b-4caa-82e0-d00f2f1c38b1-kube-api-access-nd5jj\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:02.468458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.468298    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef49c105-774b-4caa-82e0-d00f2f1c38b1-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:02.468458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:02.468306    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef49c105-774b-4caa-82e0-d00f2f1c38b1-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:03.006686 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.006647    6430 generic.go:356] "Generic (PLEG): container finished" podID="ef49c105-774b-4caa-82e0-d00f2f1c38b1" containerID="ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734" exitCode=0
Nov 03 08:41:03.006877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.006697    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"
Nov 03 08:41:03.006877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.006735    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w" event={"ID":"ef49c105-774b-4caa-82e0-d00f2f1c38b1","Type":"ContainerDied","Data":"ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734"}
Nov 03 08:41:03.006877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.006795    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w" event={"ID":"ef49c105-774b-4caa-82e0-d00f2f1c38b1","Type":"ContainerDied","Data":"bc2146c3dee4cd595e6e38d993719a539136bf949004e4b495d08d8c95c5cbae"}
Nov 03 08:41:03.006877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.006816    6430 scope.go:122] "RemoveContainer" containerID="ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734"
Nov 03 08:41:03.015471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.015453    6430 scope.go:122] "RemoveContainer" containerID="ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734"
Nov 03 08:41:03.015730 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:41:03.015714    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734\": container with ID starting with ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734 not found: ID does not exist" containerID="ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734"
Nov 03 08:41:03.015783 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.015739    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734"} err="failed to get container status \"ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734\": rpc error: code = NotFound desc = could not find container \"ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734\": container with ID starting with ad83875b4f939184ad7e5bd7d93bd9b457f4aab1098d27faf93f62a6c800a734 not found: ID does not exist"
Nov 03 08:41:03.020523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.020502    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"]
Nov 03 08:41:03.023926 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:03.023882    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-vnc7w"]
Nov 03 08:41:04.410652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:04.410621    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:04.536357 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:04.536326    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:04.622117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:04.622088    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef49c105-774b-4caa-82e0-d00f2f1c38b1" path="/var/lib/kubelet/pods/ef49c105-774b-4caa-82e0-d00f2f1c38b1/volumes"
Nov 03 08:41:05.477598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.477565    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"]
Nov 03 08:41:05.477996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.477928    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ef49c105-774b-4caa-82e0-d00f2f1c38b1" containerName="maas-api"
Nov 03 08:41:05.477996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.477939    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef49c105-774b-4caa-82e0-d00f2f1c38b1" containerName="maas-api"
Nov 03 08:41:05.477996 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.477997    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef49c105-774b-4caa-82e0-d00f2f1c38b1" containerName="maas-api"
Nov 03 08:41:05.482295 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.482277    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.484755 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.484721    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d3aeb93c-serving-cert\""
Nov 03 08:41:05.490308 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.490043    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"]
Nov 03 08:41:05.598343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.598306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-maas-api-tls\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.598523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.598396    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-metrics-certs\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.598523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.598476    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442146ee-9d81-4cea-a420-5ba2a7d4f990-openshift-service-ca\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.598523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.598494    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9m5n\" (UniqueName: \"kubernetes.io/projected/442146ee-9d81-4cea-a420-5ba2a7d4f990-kube-api-access-s9m5n\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.665522 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.665483    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:05.672299 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.672269    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"]
Nov 03 08:41:05.675824 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.675803    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:05.681471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.681445    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d3aeb93c-dockercfg-2pm6c\""
Nov 03 08:41:05.681908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.681891    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d3aeb93c\""
Nov 03 08:41:05.686224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.686202    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"]
Nov 03 08:41:05.699272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.699246    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-metrics-certs\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.699397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.699328    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442146ee-9d81-4cea-a420-5ba2a7d4f990-openshift-service-ca\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.699397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.699357    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9m5n\" (UniqueName: \"kubernetes.io/projected/442146ee-9d81-4cea-a420-5ba2a7d4f990-kube-api-access-s9m5n\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.699872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.699610    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-maas-api-tls\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.700355 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.700305    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442146ee-9d81-4cea-a420-5ba2a7d4f990-openshift-service-ca\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.702407 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.702380    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-metrics-certs\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.702513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.702436    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-maas-api-tls\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.706557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.706531    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9m5n\" (UniqueName: \"kubernetes.io/projected/442146ee-9d81-4cea-a420-5ba2a7d4f990-kube-api-access-s9m5n\") pod \"maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.747022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.746964    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"]
Nov 03 08:41:05.750650 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.750635    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:05.759574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.759550    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"]
Nov 03 08:41:05.794859 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.794805    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:05.797633 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.797608    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:05.800243 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.800216    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f177fe-3b40-4eed-a418-e0478a65f081-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") " pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:05.800381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.800266    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:05.800381 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.800349    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmq5\" (UniqueName: \"kubernetes.io/projected/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-kube-api-access-bcmq5\") pod \"payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:05.800498 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.800431    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v466s\" (UniqueName: \"kubernetes.io/projected/58f177fe-3b40-4eed-a418-e0478a65f081-kube-api-access-v466s\") pod \"payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") " pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:05.901816 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.901773    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v466s\" (UniqueName: \"kubernetes.io/projected/58f177fe-3b40-4eed-a418-e0478a65f081-kube-api-access-v466s\") pod \"payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") " pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:05.901972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.901878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f177fe-3b40-4eed-a418-e0478a65f081-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") " pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:05.901972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.901914    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:05.901972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.901947    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmq5\" (UniqueName: \"kubernetes.io/projected/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-kube-api-access-bcmq5\") pod \"payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:05.903394 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.903367    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f177fe-3b40-4eed-a418-e0478a65f081-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") " pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:05.904167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.904138    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:05.910646 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.910595    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmq5\" (UniqueName: \"kubernetes.io/projected/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-kube-api-access-bcmq5\") pod \"payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:05.912016 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.911974    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v466s\" (UniqueName: \"kubernetes.io/projected/58f177fe-3b40-4eed-a418-e0478a65f081-kube-api-access-v466s\") pod \"payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") " pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:05.989465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:05.989438    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:06.064723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:06.064689    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:06.091808 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:06.091776    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"]
Nov 03 08:41:06.263930 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:06.263739    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"]
Nov 03 08:41:06.266863 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:41:06.266838    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ddd39f_e0b2_4f74_8244_70e4ed6f3814.slice/crio-a72dda075351a3294114d51db67f7a16897e8b0234e1eab259f658aad94d5f0b WatchSource:0}: Error finding container a72dda075351a3294114d51db67f7a16897e8b0234e1eab259f658aad94d5f0b: Status 404 returned error can't find the container with id a72dda075351a3294114d51db67f7a16897e8b0234e1eab259f658aad94d5f0b
Nov 03 08:41:06.351610 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:06.351567    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"]
Nov 03 08:41:06.355205 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:06.355174    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:06.357043 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:41:06.357017    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f177fe_3b40_4eed_a418_e0478a65f081.slice/crio-4b4b748f74ecc8bcddb8e1d5b8ca44b9e3a71ff2655b2f5937255ddb9325737d WatchSource:0}: Error finding container 4b4b748f74ecc8bcddb8e1d5b8ca44b9e3a71ff2655b2f5937255ddb9325737d: Status 404 returned error can't find the container with id 4b4b748f74ecc8bcddb8e1d5b8ca44b9e3a71ff2655b2f5937255ddb9325737d
Nov 03 08:41:07.028309 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.028275    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g" event={"ID":"58f177fe-3b40-4eed-a418-e0478a65f081","Type":"ContainerStarted","Data":"f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54"}
Nov 03 08:41:07.028790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.028326    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g" event={"ID":"58f177fe-3b40-4eed-a418-e0478a65f081","Type":"ContainerStarted","Data":"4b4b748f74ecc8bcddb8e1d5b8ca44b9e3a71ff2655b2f5937255ddb9325737d"}
Nov 03 08:41:07.028790 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.028385    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:07.029982 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.029957    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf" event={"ID":"442146ee-9d81-4cea-a420-5ba2a7d4f990","Type":"ContainerStarted","Data":"fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a"}
Nov 03 08:41:07.030141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.030010    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:07.030141 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.030034    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf" event={"ID":"442146ee-9d81-4cea-a420-5ba2a7d4f990","Type":"ContainerStarted","Data":"589b5a8d161bf2f052ee09401df54e7591dd5bf4875d69f2f442c339399d37bf"}
Nov 03 08:41:07.031510 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.031485    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j" event={"ID":"16ddd39f-e0b2-4f74-8244-70e4ed6f3814","Type":"ContainerStarted","Data":"0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e"}
Nov 03 08:41:07.031590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.031529    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j" event={"ID":"16ddd39f-e0b2-4f74-8244-70e4ed6f3814","Type":"ContainerStarted","Data":"a72dda075351a3294114d51db67f7a16897e8b0234e1eab259f658aad94d5f0b"}
Nov 03 08:41:07.031590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.031566    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:07.043353 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.043314    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g" podStartSLOduration=2.043303231 podStartE2EDuration="2.043303231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:07.042188265 +0000 UTC m=+2816.545023778" watchObservedRunningTime="2026-11-03 08:41:07.043303231 +0000 UTC m=+2816.546138739"
Nov 03 08:41:07.045641 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.045622    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:07.058311 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.058271    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf" podStartSLOduration=1.643760797 podStartE2EDuration="2.058260924s" totalImagesPullingTime="414.500127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:07.057895893 +0000 UTC m=+2816.560731418" watchObservedRunningTime="2026-11-03 08:41:07.058260924 +0000 UTC m=+2816.561096437"
Nov 03 08:41:07.075460 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.075423    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j" podStartSLOduration=2.075413021 podStartE2EDuration="2.075413021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:07.074605389 +0000 UTC m=+2816.577440927" watchObservedRunningTime="2026-11-03 08:41:07.075413021 +0000 UTC m=+2816.578248536"
Nov 03 08:41:07.761129 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:07.761096    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:08.372315 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:08.372278    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:08.937964 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:08.937932    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:09.538405 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:09.538369    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:10.144781 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:10.144743    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:10.848639 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:10.848610    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:11.538537 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:11.538502    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:12.146422 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:12.146388    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:12.740288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:12.740253    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:13.042340 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:13.042261    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:13.345265 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:13.345177    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:14.013150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:14.013116    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:14.737047 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:14.737013    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:15.121811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:15.121704    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:15.340662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:15.340625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:15.937367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:15.937337    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:16.549045 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:16.549015    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:17.149857 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:17.149815    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:17.735854 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:17.735826    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:18.038326 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:18.038249    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:18.038839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:18.038817    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:18.350150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:18.350044    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:18.927198 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:18.927165    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:19.528834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:19.528801    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:20.136027 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:20.135989    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:20.851670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:20.851629    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:21.438981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:21.438948    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:22.032766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:22.032729    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:22.688592 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:22.688553    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:23.332769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:23.332733    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:23.935750 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:23.935718    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:24.634836 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:24.634802    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:25.231136 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:25.231104    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:25.883419 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:25.883390    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:26.440118 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:26.440086    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:27.030974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:27.030937    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:27.750393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:27.750361    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:28.343181 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:28.343141    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:28.378843 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:41:28.378809    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:41:28.960126 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:28.960052    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:29.557474 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:29.557443    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:30.146436 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.146404    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:30.254961 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.254925    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"]
Nov 03 08:41:30.267667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.267641    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"]
Nov 03 08:41:30.365127 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.365088    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:30.470104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.470051    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:30.653611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.653577    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"]
Nov 03 08:41:30.654202 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.653763    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf" podUID="442146ee-9d81-4cea-a420-5ba2a7d4f990" containerName="maas-api" containerID="cri-o://fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a" gracePeriod=30
Nov 03 08:41:30.730304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.730218    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:30.871948 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.871908    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:30.910906 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.910880    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:30.934449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.934415    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-metrics-certs\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-metrics-certs\") pod \"442146ee-9d81-4cea-a420-5ba2a7d4f990\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") "
Nov 03 08:41:30.934652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.934497    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442146ee-9d81-4cea-a420-5ba2a7d4f990-kube-api-access-s9m5n\" (UniqueName: \"kubernetes.io/projected/442146ee-9d81-4cea-a420-5ba2a7d4f990-kube-api-access-s9m5n\") pod \"442146ee-9d81-4cea-a420-5ba2a7d4f990\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") "
Nov 03 08:41:30.938768 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.938740    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-metrics-certs" pod "442146ee-9d81-4cea-a420-5ba2a7d4f990" (UID: "442146ee-9d81-4cea-a420-5ba2a7d4f990"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:30.944401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.944361    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442146ee-9d81-4cea-a420-5ba2a7d4f990-kube-api-access-s9m5n" pod "442146ee-9d81-4cea-a420-5ba2a7d4f990" (UID: "442146ee-9d81-4cea-a420-5ba2a7d4f990"). InnerVolumeSpecName "kube-api-access-s9m5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:31.000090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:30.999990    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:31.035104 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.035075    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442146ee-9d81-4cea-a420-5ba2a7d4f990-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442146ee-9d81-4cea-a420-5ba2a7d4f990-openshift-service-ca\") pod \"442146ee-9d81-4cea-a420-5ba2a7d4f990\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") "
Nov 03 08:41:31.035204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.035136    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-maas-api-tls\") pod \"442146ee-9d81-4cea-a420-5ba2a7d4f990\" (UID: \"442146ee-9d81-4cea-a420-5ba2a7d4f990\") "
Nov 03 08:41:31.035368 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.035358    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:31.035408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.035372    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9m5n\" (UniqueName: \"kubernetes.io/projected/442146ee-9d81-4cea-a420-5ba2a7d4f990-kube-api-access-s9m5n\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:31.035444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.035409    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442146ee-9d81-4cea-a420-5ba2a7d4f990-openshift-service-ca" pod "442146ee-9d81-4cea-a420-5ba2a7d4f990" (UID: "442146ee-9d81-4cea-a420-5ba2a7d4f990"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:31.037335 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.037316    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-maas-api-tls" pod "442146ee-9d81-4cea-a420-5ba2a7d4f990" (UID: "442146ee-9d81-4cea-a420-5ba2a7d4f990"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:31.136374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.136339    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442146ee-9d81-4cea-a420-5ba2a7d4f990-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:31.136374 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.136369    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442146ee-9d81-4cea-a420-5ba2a7d4f990-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:31.151772 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.151733    6430 generic.go:356] "Generic (PLEG): container finished" podID="442146ee-9d81-4cea-a420-5ba2a7d4f990" containerID="fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a" exitCode=0
Nov 03 08:41:31.151925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.151782    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"
Nov 03 08:41:31.151925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.151781    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf" event={"ID":"442146ee-9d81-4cea-a420-5ba2a7d4f990","Type":"ContainerDied","Data":"fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a"}
Nov 03 08:41:31.151925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.151917    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf" event={"ID":"442146ee-9d81-4cea-a420-5ba2a7d4f990","Type":"ContainerDied","Data":"589b5a8d161bf2f052ee09401df54e7591dd5bf4875d69f2f442c339399d37bf"}
Nov 03 08:41:31.152114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.151947    6430 scope.go:122] "RemoveContainer" containerID="fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a"
Nov 03 08:41:31.161951 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.161931    6430 scope.go:122] "RemoveContainer" containerID="fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a"
Nov 03 08:41:31.162266 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:41:31.162247    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a\": container with ID starting with fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a not found: ID does not exist" containerID="fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a"
Nov 03 08:41:31.162347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.162276    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a"} err="failed to get container status \"fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a\": rpc error: code = NotFound desc = could not find container \"fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a\": container with ID starting with fba3fea23575c8322c828035d2039fa52e01f2f847496a0baf6783bd9c4eac1a not found: ID does not exist"
Nov 03 08:41:31.175418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.175390    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"]
Nov 03 08:41:31.177465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:31.177444    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3aeb93c-5bbc6cd897-nvpjf"]
Nov 03 08:41:32.615547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:32.615511    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442146ee-9d81-4cea-a420-5ba2a7d4f990" path="/var/lib/kubelet/pods/442146ee-9d81-4cea-a420-5ba2a7d4f990/volumes"
Nov 03 08:41:35.255749 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.255692    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g" podUID="58f177fe-3b40-4eed-a418-e0478a65f081" containerName="payload-processing" containerID="cri-o://f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54" gracePeriod=25
Nov 03 08:41:35.268331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.268294    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j" podUID="16ddd39f-e0b2-4f74-8244-70e4ed6f3814" containerName="payload-pre-processing" containerID="cri-o://0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e" gracePeriod=25
Nov 03 08:41:35.502978 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.502956    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:35.532244 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.532224    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:35.565883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.565854    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-plugins-config-volume\") pod \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") "
Nov 03 08:41:35.566042 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.565951    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-kube-api-access-bcmq5\" (UniqueName: \"kubernetes.io/projected/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-kube-api-access-bcmq5\") pod \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\" (UID: \"16ddd39f-e0b2-4f74-8244-70e4ed6f3814\") "
Nov 03 08:41:35.566152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.566045    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f177fe-3b40-4eed-a418-e0478a65f081-kube-api-access-v466s\" (UniqueName: \"kubernetes.io/projected/58f177fe-3b40-4eed-a418-e0478a65f081-kube-api-access-v466s\") pod \"58f177fe-3b40-4eed-a418-e0478a65f081\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") "
Nov 03 08:41:35.566152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.566117    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f177fe-3b40-4eed-a418-e0478a65f081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f177fe-3b40-4eed-a418-e0478a65f081-plugins-config-volume\") pod \"58f177fe-3b40-4eed-a418-e0478a65f081\" (UID: \"58f177fe-3b40-4eed-a418-e0478a65f081\") "
Nov 03 08:41:35.566415 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.566379    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-plugins-config-volume" pod "16ddd39f-e0b2-4f74-8244-70e4ed6f3814" (UID: "16ddd39f-e0b2-4f74-8244-70e4ed6f3814"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:35.566518 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.566440    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f177fe-3b40-4eed-a418-e0478a65f081-plugins-config-volume" pod "58f177fe-3b40-4eed-a418-e0478a65f081" (UID: "58f177fe-3b40-4eed-a418-e0478a65f081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:35.568354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.568334    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f177fe-3b40-4eed-a418-e0478a65f081-kube-api-access-v466s" pod "58f177fe-3b40-4eed-a418-e0478a65f081" (UID: "58f177fe-3b40-4eed-a418-e0478a65f081"). InnerVolumeSpecName "kube-api-access-v466s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:35.568431 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.568379    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-kube-api-access-bcmq5" pod "16ddd39f-e0b2-4f74-8244-70e4ed6f3814" (UID: "16ddd39f-e0b2-4f74-8244-70e4ed6f3814"). InnerVolumeSpecName "kube-api-access-bcmq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:35.667535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.667508    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v466s\" (UniqueName: \"kubernetes.io/projected/58f177fe-3b40-4eed-a418-e0478a65f081-kube-api-access-v466s\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:35.667535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.667530    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f177fe-3b40-4eed-a418-e0478a65f081-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:35.667535 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.667540    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:35.667735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:35.667549    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcmq5\" (UniqueName: \"kubernetes.io/projected/16ddd39f-e0b2-4f74-8244-70e4ed6f3814-kube-api-access-bcmq5\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:36.175485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.175451    6430 generic.go:356] "Generic (PLEG): container finished" podID="58f177fe-3b40-4eed-a418-e0478a65f081" containerID="f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54" exitCode=0
Nov 03 08:41:36.175721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.175506    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"
Nov 03 08:41:36.175721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.175544    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g" event={"ID":"58f177fe-3b40-4eed-a418-e0478a65f081","Type":"ContainerDied","Data":"f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54"}
Nov 03 08:41:36.175721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.175648    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g" event={"ID":"58f177fe-3b40-4eed-a418-e0478a65f081","Type":"ContainerDied","Data":"4b4b748f74ecc8bcddb8e1d5b8ca44b9e3a71ff2655b2f5937255ddb9325737d"}
Nov 03 08:41:36.175721 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.175679    6430 scope.go:122] "RemoveContainer" containerID="f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54"
Nov 03 08:41:36.176811 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.176791    6430 generic.go:356] "Generic (PLEG): container finished" podID="16ddd39f-e0b2-4f74-8244-70e4ed6f3814" containerID="0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e" exitCode=0
Nov 03 08:41:36.176895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.176825    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j" event={"ID":"16ddd39f-e0b2-4f74-8244-70e4ed6f3814","Type":"ContainerDied","Data":"0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e"}
Nov 03 08:41:36.176895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.176849    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"
Nov 03 08:41:36.176895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.176876    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j" event={"ID":"16ddd39f-e0b2-4f74-8244-70e4ed6f3814","Type":"ContainerDied","Data":"a72dda075351a3294114d51db67f7a16897e8b0234e1eab259f658aad94d5f0b"}
Nov 03 08:41:36.185771 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.185758    6430 scope.go:122] "RemoveContainer" containerID="f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54"
Nov 03 08:41:36.186052 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:41:36.186029    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54\": container with ID starting with f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54 not found: ID does not exist" containerID="f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54"
Nov 03 08:41:36.186149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.186084    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54"} err="failed to get container status \"f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54\": rpc error: code = NotFound desc = could not find container \"f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54\": container with ID starting with f9cc7ec1cd836956fd48f651b4380a91d22b4bd92205e44163e43a75f8db0a54 not found: ID does not exist"
Nov 03 08:41:36.186149 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.186102    6430 scope.go:122] "RemoveContainer" containerID="0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e"
Nov 03 08:41:36.193759 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.193741    6430 scope.go:122] "RemoveContainer" containerID="0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e"
Nov 03 08:41:36.194038 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:41:36.194010    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e\": container with ID starting with 0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e not found: ID does not exist" containerID="0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e"
Nov 03 08:41:36.194128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.194041    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e"} err="failed to get container status \"0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e\": rpc error: code = NotFound desc = could not find container \"0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e\": container with ID starting with 0438cab889def7107669e96559cd2761f7263684befba85f93a55c4814c0734e not found: ID does not exist"
Nov 03 08:41:36.196204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.196184    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"]
Nov 03 08:41:36.201341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.201323    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3aeb93c-6c6d798c95-gn48g"]
Nov 03 08:41:36.211524 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.211507    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"]
Nov 03 08:41:36.218140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.218120    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3aeb93c-65f9dccd67-w4l4j"]
Nov 03 08:41:36.476418 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.476389    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:36.615527 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.615492    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ddd39f-e0b2-4f74-8244-70e4ed6f3814" path="/var/lib/kubelet/pods/16ddd39f-e0b2-4f74-8244-70e4ed6f3814/volumes"
Nov 03 08:41:36.615871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:36.615858    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f177fe-3b40-4eed-a418-e0478a65f081" path="/var/lib/kubelet/pods/58f177fe-3b40-4eed-a418-e0478a65f081/volumes"
Nov 03 08:41:37.663128 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:37.663075    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:38.028397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.028355    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"]
Nov 03 08:41:38.028931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.028906    6430 cpu_manager.go:414] "containerMap: removing container" podUID="16ddd39f-e0b2-4f74-8244-70e4ed6f3814" containerName="payload-pre-processing"
Nov 03 08:41:38.028931 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.028929    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ddd39f-e0b2-4f74-8244-70e4ed6f3814" containerName="payload-pre-processing"
Nov 03 08:41:38.029186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.028973    6430 cpu_manager.go:414] "containerMap: removing container" podUID="58f177fe-3b40-4eed-a418-e0478a65f081" containerName="payload-processing"
Nov 03 08:41:38.029186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.028979    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f177fe-3b40-4eed-a418-e0478a65f081" containerName="payload-processing"
Nov 03 08:41:38.029186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.028988    6430 cpu_manager.go:414] "containerMap: removing container" podUID="442146ee-9d81-4cea-a420-5ba2a7d4f990" containerName="maas-api"
Nov 03 08:41:38.029186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.028995    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442146ee-9d81-4cea-a420-5ba2a7d4f990" containerName="maas-api"
Nov 03 08:41:38.029186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.029107    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f177fe-3b40-4eed-a418-e0478a65f081" containerName="payload-processing"
Nov 03 08:41:38.029186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.029122    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="442146ee-9d81-4cea-a420-5ba2a7d4f990" containerName="maas-api"
Nov 03 08:41:38.029186 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.029136    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ddd39f-e0b2-4f74-8244-70e4ed6f3814" containerName="payload-pre-processing"
Nov 03 08:41:38.032529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.032508    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.039542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.039506    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-b34c73-openshift-default-dockercfg-nzbnj\""
Nov 03 08:41:38.044813 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.044789    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"]
Nov 03 08:41:38.086848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.086814    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-certs\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.086848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.086852    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-envoy\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.087096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.086882    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jvb\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-kube-api-access-92jvb\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.087096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.086959    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istiod-ca-cert\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.087096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.087013    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-socket\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.087096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.087034    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-data\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.087096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.087071    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-podinfo\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.087272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.087184    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-token\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.087272 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.087215    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-credential-socket\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.096337 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.096305    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:38.187829 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.187796    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-token\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.187850    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-credential-socket\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.187935    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-certs\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.187971    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-envoy\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188083 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188012    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jvb\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-kube-api-access-92jvb\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188039    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istiod-ca-cert\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188113    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-socket\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188151    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-data\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188180    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-podinfo\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188896    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-credential-socket\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188926    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-certs\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.188972 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.188962    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-data\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.189270 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.189015    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-socket\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.189766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.189633    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istiod-ca-cert\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.191019 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.190988    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-envoy\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.191208 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.191189    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-podinfo\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.198140 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.198055    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-token\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.198317 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.198294    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jvb\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-kube-api-access-92jvb\") pod \"e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") " pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.277519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.277482    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:38.346578 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.346487    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:38.625000 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.624923    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"]
Nov 03 08:41:38.628737 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:41:38.628712    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee9c9f4_6733_44c8_989a_abb5d9a1517e.slice/crio-033fb0dbe8e1a6badb089db6d2fc1e1471ee31616cb1e3b541c8f785a4793161 WatchSource:0}: Error finding container 033fb0dbe8e1a6badb089db6d2fc1e1471ee31616cb1e3b541c8f785a4793161: Status 404 returned error can't find the container with id 033fb0dbe8e1a6badb089db6d2fc1e1471ee31616cb1e3b541c8f785a4793161
Nov 03 08:41:38.631264 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.631233    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:41:38.631370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.631310    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:41:38.631370 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:38.631355    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:41:39.196046 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:39.196011    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" event={"ID":"6ee9c9f4-6733-44c8-989a-abb5d9a1517e","Type":"ContainerStarted","Data":"beb20f748a025b9e0885988a55f563c5d6a0cedc1be8049377beaee89b324cff"}
Nov 03 08:41:39.196544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:39.196081    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" event={"ID":"6ee9c9f4-6733-44c8-989a-abb5d9a1517e","Type":"ContainerStarted","Data":"033fb0dbe8e1a6badb089db6d2fc1e1471ee31616cb1e3b541c8f785a4793161"}
Nov 03 08:41:39.213416 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:39.213362    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podStartSLOduration=1.2133439400000001 podStartE2EDuration="1.21334394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:39.212593165 +0000 UTC m=+2848.715428679" watchObservedRunningTime="2026-11-03 08:41:39.21334394 +0000 UTC m=+2848.716179468"
Nov 03 08:41:39.347304 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:39.347274    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:39.352456 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:39.352425    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:40.200883 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:40.200850    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:40.201908 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:40.201885    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:41.369834 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:41.369802    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:41.584324 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:41.584288    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:42.464659 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.464625    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"]
Nov 03 08:41:42.468472 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.468446    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.479839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.479809    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-b34c73-serving-cert\""
Nov 03 08:41:42.481352 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.481325    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"]
Nov 03 08:41:42.526470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.526429    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.526470 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.526473    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2bw\" (UniqueName: \"kubernetes.io/projected/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-kube-api-access-8f2bw\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.526724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.526547    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.526724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.526611    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.556402 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.556361    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:42.627110 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.627051    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.627263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.627130    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2bw\" (UniqueName: \"kubernetes.io/projected/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-kube-api-access-8f2bw\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.627263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.627162    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.627263 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.627236    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.627935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.627885    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.629706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.629685    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.629818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.629738    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.636502 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.636480    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2bw\" (UniqueName: \"kubernetes.io/projected/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-kube-api-access-8f2bw\") pod \"maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.641277 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.641255    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"]
Nov 03 08:41:42.644849 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.644826    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.652442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.652405    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-b34c73-dockercfg-sfr6v\""
Nov 03 08:41:42.652442 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.652413    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-b34c73\""
Nov 03 08:41:42.654209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.654187    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"]
Nov 03 08:41:42.677519 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.677492    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"]
Nov 03 08:41:42.680877 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.680861    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:42.685103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.685075    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:42.691037 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.691013    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"]
Nov 03 08:41:42.728504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.728434    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a146bc62-fe6f-436a-bdb4-8128ee84731f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:42.728504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.728467    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz242\" (UniqueName: \"kubernetes.io/projected/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-kube-api-access-fz242\") pod \"payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.728667 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.728624    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.728706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.728696    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952tz\" (UniqueName: \"kubernetes.io/projected/a146bc62-fe6f-436a-bdb4-8128ee84731f-kube-api-access-952tz\") pod \"payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:42.785977 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.785941    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:42.830841 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.830800    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.831017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.830865    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952tz\" (UniqueName: \"kubernetes.io/projected/a146bc62-fe6f-436a-bdb4-8128ee84731f-kube-api-access-952tz\") pod \"payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:42.831017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.830918    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a146bc62-fe6f-436a-bdb4-8128ee84731f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:42.831017 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.830954    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz242\" (UniqueName: \"kubernetes.io/projected/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-kube-api-access-fz242\") pod \"payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.832211 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.832186    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.833201 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.833173    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a146bc62-fe6f-436a-bdb4-8128ee84731f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:42.839013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.838976    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz242\" (UniqueName: \"kubernetes.io/projected/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-kube-api-access-fz242\") pod \"payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.853945 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.852293    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952tz\" (UniqueName: \"kubernetes.io/projected/a146bc62-fe6f-436a-bdb4-8128ee84731f-kube-api-access-952tz\") pod \"payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:42.958189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.958157    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:42.994723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:42.994641    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:43.088517 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:43.088386    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"]
Nov 03 08:41:43.218577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:43.218542    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh" event={"ID":"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13","Type":"ContainerStarted","Data":"b81cfaed816920089619efcf18f57c44bec7ffecf57f5960ec9f4fa4e8b1baa6"}
Nov 03 08:41:43.232150 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:43.232122    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"]
Nov 03 08:41:43.235802 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:41:43.235778    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e2d7cb_e0d5_4b17_ba00_90bed359aa45.slice/crio-72b7adeae6c99a9e8e47fd2e2c2e2f8851346b7e35273e70a6980bdf0ff2e9b3 WatchSource:0}: Error finding container 72b7adeae6c99a9e8e47fd2e2c2e2f8851346b7e35273e70a6980bdf0ff2e9b3: Status 404 returned error can't find the container with id 72b7adeae6c99a9e8e47fd2e2c2e2f8851346b7e35273e70a6980bdf0ff2e9b3
Nov 03 08:41:43.244437 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:43.244414    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:43.280807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:43.280761    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"]
Nov 03 08:41:43.284249 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:41:43.284216    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda146bc62_fe6f_436a_bdb4_8128ee84731f.slice/crio-54b45c8f6868a3cd3b865f9af9d80aa4a889937e14be1579feb328cd9fc4ec8f WatchSource:0}: Error finding container 54b45c8f6868a3cd3b865f9af9d80aa4a889937e14be1579feb328cd9fc4ec8f: Status 404 returned error can't find the container with id 54b45c8f6868a3cd3b865f9af9d80aa4a889937e14be1579feb328cd9fc4ec8f
Nov 03 08:41:43.951207 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:43.951172    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:44.227250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.227148    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs" event={"ID":"a146bc62-fe6f-436a-bdb4-8128ee84731f","Type":"ContainerStarted","Data":"a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574"}
Nov 03 08:41:44.227250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.227210    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs" event={"ID":"a146bc62-fe6f-436a-bdb4-8128ee84731f","Type":"ContainerStarted","Data":"54b45c8f6868a3cd3b865f9af9d80aa4a889937e14be1579feb328cd9fc4ec8f"}
Nov 03 08:41:44.227500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.227347    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:44.229097 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.229041    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8" event={"ID":"07e2d7cb-e0d5-4b17-ba00-90bed359aa45","Type":"ContainerStarted","Data":"4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e"}
Nov 03 08:41:44.229250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.229116    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:44.229250 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.229135    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8" event={"ID":"07e2d7cb-e0d5-4b17-ba00-90bed359aa45","Type":"ContainerStarted","Data":"72b7adeae6c99a9e8e47fd2e2c2e2f8851346b7e35273e70a6980bdf0ff2e9b3"}
Nov 03 08:41:44.230683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.230662    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh" event={"ID":"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13","Type":"ContainerStarted","Data":"48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b"}
Nov 03 08:41:44.230807 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.230711    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:44.242361 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.242316    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs" podStartSLOduration=2.242300147 podStartE2EDuration="2.242300147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:44.24225709 +0000 UTC m=+2853.745092606" watchObservedRunningTime="2026-11-03 08:41:44.242300147 +0000 UTC m=+2853.745135659"
Nov 03 08:41:44.257942 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.257886    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh" podStartSLOduration=1.7651637839999998 podStartE2EDuration="2.257872347s" totalImagesPullingTime="492.708563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:44.257301117 +0000 UTC m=+2853.760136633" watchObservedRunningTime="2026-11-03 08:41:44.257872347 +0000 UTC m=+2853.760707877"
Nov 03 08:41:44.276769 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.276710    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8" podStartSLOduration=2.276695624 podStartE2EDuration="2.276695624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:44.275049143 +0000 UTC m=+2853.777884657" watchObservedRunningTime="2026-11-03 08:41:44.276695624 +0000 UTC m=+2853.779531138"
Nov 03 08:41:44.691547 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:44.691510    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:45.649224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:45.649190    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:46.356955 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:46.356919    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:47.077163 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:47.077130    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:47.768598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:47.768566    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:48.375953 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:48.375900    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:49.042235 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:49.042199    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:49.788369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:49.788333    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:50.241329 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:50.241298    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:41:50.487872 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:50.487840    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:51.063855 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:51.063812    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:51.706555 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:51.706516    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:52.442706 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:52.442665    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:53.053500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:53.053459    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:53.767523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:53.767485    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:54.283280 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:54.283249    6430 generic.go:356] "Generic (PLEG): container finished" podID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerID="beb20f748a025b9e0885988a55f563c5d6a0cedc1be8049377beaee89b324cff" exitCode=0
Nov 03 08:41:54.283462 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:54.283314    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" event={"ID":"6ee9c9f4-6733-44c8-989a-abb5d9a1517e","Type":"ContainerDied","Data":"beb20f748a025b9e0885988a55f563c5d6a0cedc1be8049377beaee89b324cff"}
Nov 03 08:41:54.283682 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:54.283667    6430 scope.go:122] "RemoveContainer" containerID="beb20f748a025b9e0885988a55f563c5d6a0cedc1be8049377beaee89b324cff"
Nov 03 08:41:54.284924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:54.284896    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:41:54.285092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:54.284981    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:41:54.285092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:54.285028    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 08:41:54.352780 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:54.352748    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:55.027504 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.027461    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:55.214086 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.210165    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:55.239189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.239159    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:41:55.239371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.239231    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:41:55.290089 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.289986    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" event={"ID":"6ee9c9f4-6733-44c8-989a-abb5d9a1517e","Type":"ContainerStarted","Data":"e8b4bf54bd199eb99fc5085d8085bb8c40f3cd1865c6900fc08680188e19cd51"}
Nov 03 08:41:55.347275 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.347248    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:41:55.348367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.348339    6430 patch_prober.go:29] interesting pod/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:41:55.348521 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.348396    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 08:41:55.752661 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:55.752626    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:56.341371 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:56.341332    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:56.347386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:56.347360    6430 patch_prober.go:29] interesting pod/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:41:56.347525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:56.347407    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 08:41:56.532733 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:56.532698    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:57.282523 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:57.282488    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:57.347974 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:57.347940    6430 patch_prober.go:29] interesting pod/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:41:57.348412 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:57.347991    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 08:41:58.074598 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:58.074560    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:58.236739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:58.236705    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:58.347876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:58.347790    6430 patch_prober.go:29] interesting pod/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:41:58.347876 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:58.347834    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 08:41:58.374867 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:41:58.374834    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:41:58.861018 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:58.860970    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:41:59.349568 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:59.349533    6430 patch_prober.go:29] interesting pod/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:41:59.350138 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:59.349588    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 08:41:59.547917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:41:59.547865    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:00.160224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:00.160188    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:00.346799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:00.346762    6430 patch_prober.go:29] interesting pod/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:42:00.347005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:00.346820    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 08:42:00.741624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:00.741587    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:01.352230 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:01.352197    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:42:01.352576 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:01.352556    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:42:01.353209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:01.353185    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:42:01.449699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:01.449662    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:02.046653 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:02.046616    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:02.637022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:02.636980    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:03.337212 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:03.337177    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:03.951109 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:03.951048    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:04.582875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:04.582840    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:05.156973 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:05.156940    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:05.746396 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:05.746360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:06.346388 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:06.346357    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:07.033301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:07.033268    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:07.638342 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:07.638302    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:08.250895 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:08.250861    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:08.838391 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:08.838360    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:09.441880 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:09.441848    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:10.051116 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:10.051049    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:10.720921 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:10.720891    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:11.356446 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:11.356403    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:11.957704 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:11.957676    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:12.550051 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:12.550020    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:13.139734 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:13.139704    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:13.832471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:13.832432    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:14.440557 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:14.440524    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:15.054709 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:15.054673    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:15.180724 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:15.180694    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:15.667237 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:15.667173    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:16.345154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:16.345120    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:16.946701 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:16.946662    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:17.544152 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:17.544115    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:18.144685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:18.144652    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:18.747558 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:18.747519    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:19.447914 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:19.447824    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:20.037297 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:20.037260    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:20.644351 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:20.644283    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:21.244033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:21.243986    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:21.838033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:21.837995    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:22.432729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:22.432697    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:23.137672 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:23.137631    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:23.733991 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:23.733958    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:24.333503 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:24.333473    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:24.932778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:24.932750    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:25.536241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:25.536212    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:26.138695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:26.138637    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:26.735477 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:26.735431    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:27.330033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:27.329994    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:27.939458 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:27.939394    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:28.375034 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:42:28.374953    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:42:28.528692 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:28.528647    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:29.139399 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:29.139362    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:29.820390 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:29.820350    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:30.446429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:30.446371    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:30.850542 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:30.850463    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:31.053397 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:31.053365    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:31.647273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:31.647242    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:32.312490 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:32.312446    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:32.935910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:32.935866    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:33.551573 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:33.551527    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:34.157056 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:34.157018    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:34.746886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:34.746836    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:35.340747 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:35.340712    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:35.938005 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:35.937973    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:36.602600 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:36.602567    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:37.133221 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:37.133191    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:37.730485 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:37.730444    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:38.335154 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:38.335117    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:38.936622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:38.936584    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:39.533762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:39.533728    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:40.123258 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:40.123219    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:40.807254 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:40.807204    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:41.447935 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:41.447893    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:42.042178 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:42.042136    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:42.636162 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:42.636128    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:43.230484 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:43.230450    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:43.835534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:43.835502    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:44.019662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:44.019623    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"]
Nov 03 08:42:44.021617 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:44.021591    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l8zz8"]
Nov 03 08:42:44.431032 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:44.430994    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:44.615698 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:44.615656    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a" path="/var/lib/kubelet/pods/c796c424-6c1a-43b1-bd5e-ad9e9cd76d9a/volumes"
Nov 03 08:42:45.036486 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:45.036453    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:45.635029 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:45.634993    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:46.235636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:46.235605    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:46.944350 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:46.944320    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:47.549393 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:47.549349    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:48.263529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:48.263490    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:48.839992 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:48.839948    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:49.441359 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:49.441322    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:50.054423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:50.054390    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:50.716131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:50.716089    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:51.358665 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:51.358625    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:51.950820 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:51.950788    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:52.543318 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:52.543275    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:53.147613 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:53.147571    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:53.831429 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:53.831390    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:54.431618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:54.431577    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:55.043685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:55.043648    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:55.642904 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:55.642869    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:56.236765 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:56.236714    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:56.846967 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:56.846937    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:57.493210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:57.493178    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:58.044818 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:58.044782    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:58.375282 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:42:58.375204    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:42:58.708084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:58.708042    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:59.239740 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:59.239687    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:42:59.886684 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:42:59.886651    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:00.429209 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:00.429171    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:01.045375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:01.045341    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:01.646114 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:01.646055    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:02.241652 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:02.241617    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:02.942241 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:02.942210    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:03.543169 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:03.543136    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:04.141832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:04.141794    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:04.729591 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:04.729553    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:05.021929 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:05.021819    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"]
Nov 03 08:43:05.023760 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:05.023726    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vlfkx"]
Nov 03 08:43:05.335441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:05.335349    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:05.920924 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:05.920893    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:06.565288 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:06.565257    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:06.615799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:06.615770    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788c88e6-6590-4317-b1db-922f79ff46af" path="/var/lib/kubelet/pods/788c88e6-6590-4317-b1db-922f79ff46af/volumes"
Nov 03 08:43:07.130890 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:07.130847    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:07.821408 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:07.821378    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:08.421687 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:08.421649    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:09.029744 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:09.029687    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:09.617466 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:09.617427    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:10.208403 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:10.208350    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:10.831096 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:10.831045    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:11.439648 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:11.439611    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:12.032310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:12.032271    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:12.629449 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:12.629418    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:13.227577 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:13.227546    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:13.821712 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:13.821682    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:14.418287 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:14.418250    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:15.046743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:15.046711    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:15.206341 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:15.206304    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:15.636869 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:15.636832    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:16.441283 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:16.441248    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:17.055675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:17.055646    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:17.746879 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:17.746851    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:18.706937 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:18.706901    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:18.923347 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:18.923310    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:19.531886 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:19.531837    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:20.232425 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:20.232389    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:20.915354 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:20.915324    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:21.445078 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:21.445026    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:22.039993 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:22.039959    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:22.633735 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:22.633702    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:23.233865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:23.233834    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:23.826216 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:23.826177    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:23.997310 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:23.997270    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:24.444271 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:24.444240    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:25.096640 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:25.096605    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:25.644233 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:25.644198    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:26.045867 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:26.045829    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"]
Nov 03 08:43:26.058636 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:26.058602    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"]
Nov 03 08:43:26.144662 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:26.144622    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:26.345471 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:26.345367    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:26.465513 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:26.465480    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:28.375350 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:43:28.375314    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:43:31.046525 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.046468    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs" podUID="a146bc62-fe6f-436a-bdb4-8128ee84731f" containerName="payload-processing" containerID="cri-o://a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574" gracePeriod=25
Nov 03 08:43:31.058917 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.058866    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8" podUID="07e2d7cb-e0d5-4b17-ba00-90bed359aa45" containerName="payload-pre-processing" containerID="cri-o://4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e" gracePeriod=25
Nov 03 08:43:31.302257 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.302202    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:43:31.330482 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.330459    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:43:31.404562 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.404534    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a146bc62-fe6f-436a-bdb4-8128ee84731f-kube-api-access-952tz\" (UniqueName: \"kubernetes.io/projected/a146bc62-fe6f-436a-bdb4-8128ee84731f-kube-api-access-952tz\") pod \"a146bc62-fe6f-436a-bdb4-8128ee84731f\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") "
Nov 03 08:43:31.404694 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.404624    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a146bc62-fe6f-436a-bdb4-8128ee84731f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a146bc62-fe6f-436a-bdb4-8128ee84731f-plugins-config-volume\") pod \"a146bc62-fe6f-436a-bdb4-8128ee84731f\" (UID: \"a146bc62-fe6f-436a-bdb4-8128ee84731f\") "
Nov 03 08:43:31.404995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.404975    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a146bc62-fe6f-436a-bdb4-8128ee84731f-plugins-config-volume" pod "a146bc62-fe6f-436a-bdb4-8128ee84731f" (UID: "a146bc62-fe6f-436a-bdb4-8128ee84731f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:43:31.406848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.406830    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a146bc62-fe6f-436a-bdb4-8128ee84731f-kube-api-access-952tz" pod "a146bc62-fe6f-436a-bdb4-8128ee84731f" (UID: "a146bc62-fe6f-436a-bdb4-8128ee84731f"). InnerVolumeSpecName "kube-api-access-952tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:43:31.505430 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.505404    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-kube-api-access-fz242\" (UniqueName: \"kubernetes.io/projected/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-kube-api-access-fz242\") pod \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") "
Nov 03 08:43:31.505532 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.505500    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-plugins-config-volume\") pod \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\" (UID: \"07e2d7cb-e0d5-4b17-ba00-90bed359aa45\") "
Nov 03 08:43:31.505685 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.505674    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a146bc62-fe6f-436a-bdb4-8128ee84731f-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:31.505723 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.505688    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952tz\" (UniqueName: \"kubernetes.io/projected/a146bc62-fe6f-436a-bdb4-8128ee84731f-kube-api-access-952tz\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:31.505832 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.505815    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-plugins-config-volume" pod "07e2d7cb-e0d5-4b17-ba00-90bed359aa45" (UID: "07e2d7cb-e0d5-4b17-ba00-90bed359aa45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:43:31.507505 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.507486    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-kube-api-access-fz242" pod "07e2d7cb-e0d5-4b17-ba00-90bed359aa45" (UID: "07e2d7cb-e0d5-4b17-ba00-90bed359aa45"). InnerVolumeSpecName "kube-api-access-fz242". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:43:31.607090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.607017    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz242\" (UniqueName: \"kubernetes.io/projected/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-kube-api-access-fz242\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:31.607090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.607039    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2d7cb-e0d5-4b17-ba00-90bed359aa45-plugins-config-volume\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:31.717956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.717920    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"]
Nov 03 08:43:31.718210 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.718182    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh" podUID="0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" containerName="maas-api" containerID="cri-o://48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b" gracePeriod=30
Nov 03 08:43:31.778228 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.778195    6430 generic.go:356] "Generic (PLEG): container finished" podID="a146bc62-fe6f-436a-bdb4-8128ee84731f" containerID="a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574" exitCode=0
Nov 03 08:43:31.778440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.778244    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"
Nov 03 08:43:31.778440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.778279    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs" event={"ID":"a146bc62-fe6f-436a-bdb4-8128ee84731f","Type":"ContainerDied","Data":"a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574"}
Nov 03 08:43:31.778440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.778337    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs" event={"ID":"a146bc62-fe6f-436a-bdb4-8128ee84731f","Type":"ContainerDied","Data":"54b45c8f6868a3cd3b865f9af9d80aa4a889937e14be1579feb328cd9fc4ec8f"}
Nov 03 08:43:31.778440 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.778368    6430 scope.go:122] "RemoveContainer" containerID="a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574"
Nov 03 08:43:31.779990 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.779965    6430 generic.go:356] "Generic (PLEG): container finished" podID="07e2d7cb-e0d5-4b17-ba00-90bed359aa45" containerID="4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e" exitCode=0
Nov 03 08:43:31.780131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.780018    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8" event={"ID":"07e2d7cb-e0d5-4b17-ba00-90bed359aa45","Type":"ContainerDied","Data":"4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e"}
Nov 03 08:43:31.780131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.780036    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"
Nov 03 08:43:31.780131 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.780078    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8" event={"ID":"07e2d7cb-e0d5-4b17-ba00-90bed359aa45","Type":"ContainerDied","Data":"72b7adeae6c99a9e8e47fd2e2c2e2f8851346b7e35273e70a6980bdf0ff2e9b3"}
Nov 03 08:43:31.800180 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.800143    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:31.804013 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.803985    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"]
Nov 03 08:43:31.809566 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.809536    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b34c73-596dd8bbc9-knbs8"]
Nov 03 08:43:31.820050 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.820023    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"]
Nov 03 08:43:31.823618 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.823582    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b34c73-8594699cf-mszgs"]
Nov 03 08:43:31.830214 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.830189    6430 scope.go:122] "RemoveContainer" containerID="a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574"
Nov 03 08:43:31.830537 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:43:31.830516    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574\": container with ID starting with a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574 not found: ID does not exist" containerID="a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574"
Nov 03 08:43:31.830623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.830543    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574"} err="failed to get container status \"a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574\": rpc error: code = NotFound desc = could not find container \"a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574\": container with ID starting with a2dce164514c902046bf07c2044683daa1ec0c95185a42df8956fd31b1974574 not found: ID does not exist"
Nov 03 08:43:31.830623 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.830559    6430 scope.go:122] "RemoveContainer" containerID="4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e"
Nov 03 08:43:31.838500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.838477    6430 scope.go:122] "RemoveContainer" containerID="4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e"
Nov 03 08:43:31.838763 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:43:31.838740    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e\": container with ID starting with 4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e not found: ID does not exist" containerID="4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e"
Nov 03 08:43:31.838843 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.838774    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e"} err="failed to get container status \"4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e\": rpc error: code = NotFound desc = could not find container \"4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e\": container with ID starting with 4a1164f96e4a50c75288e1e438f65050722014803507bd9ef22b260662dc283e not found: ID does not exist"
Nov 03 08:43:31.942572 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.942531    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:31.961779 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:31.961745    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:43:32.055362 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.055304    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:32.112184 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.112149    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-metrics-certs\") pod \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") "
Nov 03 08:43:32.112377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.112207    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-openshift-service-ca\") pod \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") "
Nov 03 08:43:32.112377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.112238    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-maas-api-tls\") pod \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") "
Nov 03 08:43:32.112377 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.112288    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-kube-api-access-8f2bw\" (UniqueName: \"kubernetes.io/projected/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-kube-api-access-8f2bw\") pod \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\" (UID: \"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13\") "
Nov 03 08:43:32.112624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.112594    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-openshift-service-ca" pod "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" (UID: "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:43:32.112775 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.112719    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-openshift-service-ca\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:32.114544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.114522    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-maas-api-tls" pod "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" (UID: "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:43:32.114544 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.114530    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-kube-api-access-8f2bw" pod "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" (UID: "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13"). InnerVolumeSpecName "kube-api-access-8f2bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:43:32.114715 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.114587    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-metrics-certs" pod "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" (UID: "0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:43:32.213589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.213558    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-metrics-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:32.213729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.213591    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-maas-api-tls\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:32.213729 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.213607    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f2bw\" (UniqueName: \"kubernetes.io/projected/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13-kube-api-access-8f2bw\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:32.482092 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.481996    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:32.617401 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.617375    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e2d7cb-e0d5-4b17-ba00-90bed359aa45" path="/var/lib/kubelet/pods/07e2d7cb-e0d5-4b17-ba00-90bed359aa45/volumes"
Nov 03 08:43:32.617726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.617713    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a146bc62-fe6f-436a-bdb4-8128ee84731f" path="/var/lib/kubelet/pods/a146bc62-fe6f-436a-bdb4-8128ee84731f/volumes"
Nov 03 08:43:32.786901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.786831    6430 generic.go:356] "Generic (PLEG): container finished" podID="0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" containerID="48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b" exitCode=0
Nov 03 08:43:32.786901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.786864    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh" event={"ID":"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13","Type":"ContainerDied","Data":"48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b"}
Nov 03 08:43:32.786901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.786880    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"
Nov 03 08:43:32.786901 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.786899    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh" event={"ID":"0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13","Type":"ContainerDied","Data":"b81cfaed816920089619efcf18f57c44bec7ffecf57f5960ec9f4fa4e8b1baa6"}
Nov 03 08:43:32.787224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.786922    6430 scope.go:122] "RemoveContainer" containerID="48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b"
Nov 03 08:43:32.795910 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.795892    6430 scope.go:122] "RemoveContainer" containerID="48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b"
Nov 03 08:43:32.796187 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:43:32.796168    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b\": container with ID starting with 48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b not found: ID does not exist" containerID="48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b"
Nov 03 08:43:32.796252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.796195    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b"} err="failed to get container status \"48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b\": rpc error: code = NotFound desc = could not find container \"48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b\": container with ID starting with 48e476f95e9b43791f93e2e1b54290a925801da205fd26d8a3275804b84c750b not found: ID does not exist"
Nov 03 08:43:32.804473 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.804451    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"]
Nov 03 08:43:32.807675 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:32.807655    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b34c73-844887d968-6pfhh"]
Nov 03 08:43:34.616253 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:34.616215    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" path="/var/lib/kubelet/pods/0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13/volumes"
Nov 03 08:43:38.237052 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:38.237016    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:38.361423 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:38.361390    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:38.628875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:38.628802    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"]
Nov 03 08:43:38.629087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:38.629030    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy" containerID="cri-o://e8b4bf54bd199eb99fc5085d8085bb8c40f3cd1865c6900fc08680188e19cd51" gracePeriod=30
Nov 03 08:43:38.681690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:38.681661    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:43:43.870171 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.870137    6430 generic.go:356] "Generic (PLEG): container finished" podID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerID="e8b4bf54bd199eb99fc5085d8085bb8c40f3cd1865c6900fc08680188e19cd51" exitCode=0
Nov 03 08:43:43.870589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.870189    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" event={"ID":"6ee9c9f4-6733-44c8-989a-abb5d9a1517e","Type":"ContainerDied","Data":"e8b4bf54bd199eb99fc5085d8085bb8c40f3cd1865c6900fc08680188e19cd51"}
Nov 03 08:43:43.870589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.870245    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc" event={"ID":"6ee9c9f4-6733-44c8-989a-abb5d9a1517e","Type":"ContainerDied","Data":"033fb0dbe8e1a6badb089db6d2fc1e1471ee31616cb1e3b541c8f785a4793161"}
Nov 03 08:43:43.870589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.870262    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033fb0dbe8e1a6badb089db6d2fc1e1471ee31616cb1e3b541c8f785a4793161"
Nov 03 08:43:43.870589 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.870276    6430 scope.go:122] "RemoveContainer" containerID="beb20f748a025b9e0885988a55f563c5d6a0cedc1be8049377beaee89b324cff"
Nov 03 08:43:43.881192 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.881170    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:43:43.949690 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949660    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-certs\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.949882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949703    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-kube-api-access-92jvb\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-kube-api-access-92jvb\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.949882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949727    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-podinfo\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.949882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949758    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-credential-socket\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.949882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949786    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-socket\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.949882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949808    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-data\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.949882 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949867    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-envoy\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.950267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949895    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-token\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.950267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.949920    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istiod-ca-cert\") pod \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\" (UID: \"6ee9c9f4-6733-44c8-989a-abb5d9a1517e\") "
Nov 03 08:43:43.950267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950051    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-certs" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:43:43.950267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950172    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-credential-socket" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:43:43.950267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950190    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-socket" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:43:43.950267 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950258    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-data" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:43:43.950611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950334    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-certs\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:43.950611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950352    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-credential-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:43.950611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950365    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-workload-socket\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:43.950611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950381    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-data\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:43.950611 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.950541    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istiod-ca-cert" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:43:43.952349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.952319    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-kube-api-access-92jvb" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "kube-api-access-92jvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:43:43.952349 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.952331    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-podinfo" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:43:43.952884 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.952862    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-envoy" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:43:43.952976 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:43.952883    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-token" pod "6ee9c9f4-6733-44c8-989a-abb5d9a1517e" (UID: "6ee9c9f4-6733-44c8-989a-abb5d9a1517e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:43:44.051534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.051493    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92jvb\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-kube-api-access-92jvb\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:44.051534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.051530    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-podinfo\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:44.051534 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.051542    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-envoy\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:44.051799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.051551    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istio-token\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:44.051799 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.051560    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9c9f4-6733-44c8-989a-abb5d9a1517e-istiod-ca-cert\") on node \"ip-10-0-140-232.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:44.875900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.875821    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"
Nov 03 08:43:44.904918 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.904876    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"]
Nov 03 08:43:44.909204 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:44.909103    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b34c73-openshift-default-7b9979fcd9-54ffc"]
Nov 03 08:43:46.617369 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:43:46.617335    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" path="/var/lib/kubelet/pods/6ee9c9f4-6733-44c8-989a-abb5d9a1517e/volumes"
Nov 03 08:43:58.374628 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:43:58.374546    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:44:13.008963 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:13.008929    6430 scope.go:122] "RemoveContainer" containerID="53b9e66380b44ad0eec2a89555a8f9a7dae4db93b86bbf5f252c80987b179ff8"
Nov 03 08:44:13.017386 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:13.017364    6430 scope.go:122] "RemoveContainer" containerID="7863ca21b36598886a3032f8ff72c94f95fc2c2fef5790bb88a43f2c932f7f28"
Nov 03 08:44:15.267033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:15.266986    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:44:28.392845 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:44:28.392787    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:44:38.287182 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:38.287145    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-v6gdq_35013bdc-2709-4435-8e0f-cbb535870313/manager/0.log"
Nov 03 08:44:38.471468 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:38.471431    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p6rtl_5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1/manager/0.log"
Nov 03 08:44:38.661273 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:38.661203    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-r57t2_862e4682-de0b-4e34-9bb8-626f0ab19a83/manager/0.log"
Nov 03 08:44:38.906343 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:38.906307    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799454fcf4-2rncm_25c36014-1407-4536-ad79-b3b4e16be4c0/manager/0.log"
Nov 03 08:44:40.553933 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:40.553906    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f6d4c9f4b-wkd89_f9508d41-956b-49ec-8306-a1ea83904f0b/maas-api/0.log"
Nov 03 08:44:41.378227 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:41.378194    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a1bd9efb-j5wxd_e8326ebd-5944-4248-9ca3-cd5c5ba470ac/revoke-keys/0.log"
Nov 03 08:44:41.732831 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:41.732802    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-t88sq_21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe/postgres/0.log"
Nov 03 08:44:43.568495 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:43.568466    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-q2k2f_145aee33-7914-4590-bc10-6c79746d247e/manager/0.log"
Nov 03 08:44:43.727766 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:43.727732    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qp7tx_3c8d6356-3592-4952-846d-3b7c992b6a3c/kuadrant-console-plugin/0.log"
Nov 03 08:44:45.599822 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:45.599793    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-2hhrl_e23318d3-5990-4aea-9c49-e821bb20fd56/kube-auth-proxy/0.log"
Nov 03 08:44:45.943695 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:45.943666    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-66kgk_7a5a8bac-1e6a-4fb8-9852-abc001854117/istio-proxy/0.log"
Nov 03 08:44:46.472632 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:46.472604    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bchrg_f2b07e41-a95f-452b-b166-c4532a2daee0/istio-proxy/0.log"
Nov 03 08:44:47.182670 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:47.182637    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-s92vd_6ef2358a-4551-4237-bed0-4ad8b79652bd/istio-proxy/0.log"
Nov 03 08:44:58.373921 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:44:58.373878    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:44:59.838179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:44:59.838152    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-v6gdq_35013bdc-2709-4435-8e0f-cbb535870313/manager/0.log"
Nov 03 08:45:00.018331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:00.018303    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p6rtl_5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1/manager/0.log"
Nov 03 08:45:00.204336 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:00.204288    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-r57t2_862e4682-de0b-4e34-9bb8-626f0ab19a83/manager/0.log"
Nov 03 08:45:00.414090 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:00.414042    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799454fcf4-2rncm_25c36014-1407-4536-ad79-b3b4e16be4c0/manager/0.log"
Nov 03 08:45:01.658958 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:01.658932    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f6d4c9f4b-wkd89_f9508d41-956b-49ec-8306-a1ea83904f0b/maas-api/0.log"
Nov 03 08:45:02.660785 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:02.660759    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a1bd9efb-j5wxd_e8326ebd-5944-4248-9ca3-cd5c5ba470ac/revoke-keys/0.log"
Nov 03 08:45:03.012914 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:03.012886    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-t88sq_21b7eca9-6d1d-49be-a7e7-48d27ccfb8fe/postgres/0.log"
Nov 03 08:45:04.351835 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:04.351802    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:45:04.450683 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:04.450660    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-q2k2f_145aee33-7914-4590-bc10-6c79746d247e/manager/0.log"
Nov 03 08:45:04.613981 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:04.613912    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qp7tx_3c8d6356-3592-4952-846d-3b7c992b6a3c/kuadrant-console-plugin/0.log"
Nov 03 08:45:06.025594 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:06.025563    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-2hhrl_e23318d3-5990-4aea-9c49-e821bb20fd56/kube-auth-proxy/0.log"
Nov 03 08:45:06.360742 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:06.360652    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-66kgk_7a5a8bac-1e6a-4fb8-9852-abc001854117/istio-proxy/0.log"
Nov 03 08:45:06.892739 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:06.892708    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bchrg_f2b07e41-a95f-452b-b166-c4532a2daee0/istio-proxy/0.log"
Nov 03 08:45:07.583438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:07.583405    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-s92vd_6ef2358a-4551-4237-bed0-4ad8b79652bd/istio-proxy/0.log"
Nov 03 08:45:15.319438 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:15.319395    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:45:28.374653 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:45:28.374611    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:45:43.227624 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:43.227580    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799454fcf4-2rncm_25c36014-1407-4536-ad79-b3b4e16be4c0/manager/0.log"
Nov 03 08:45:43.481839 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:43.481748    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799454fcf4-2rncm_25c36014-1407-4536-ad79-b3b4e16be4c0/manager/0.log"
Nov 03 08:45:46.020095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:46.020041    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"]
Nov 03 08:45:46.025156 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:46.025127    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bd9efb-j5wxd"]
Nov 03 08:45:46.614994 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:45:46.614963    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8326ebd-5944-4248-9ca3-cd5c5ba470ac" path="/var/lib/kubelet/pods/e8326ebd-5944-4248-9ca3-cd5c5ba470ac/volumes"
Nov 03 08:45:58.374001 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:45:58.373951    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:46:16.749465 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:16.749426    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:46:25.509956 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:25.509901    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799454fcf4-2rncm_25c36014-1407-4536-ad79-b3b4e16be4c0/manager/0.log"
Nov 03 08:46:25.766224 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:25.766141    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799454fcf4-2rncm_25c36014-1407-4536-ad79-b3b4e16be4c0/manager/0.log"
Nov 03 08:46:26.897943 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.897912    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lsk8v/must-gather-dtd55"]
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898297    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" containerName="maas-api"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898309    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" containerName="maas-api"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898321    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898330    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898338    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898343    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898349    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a146bc62-fe6f-436a-bdb4-8128ee84731f" containerName="payload-processing"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898355    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a146bc62-fe6f-436a-bdb4-8128ee84731f" containerName="payload-processing"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898365    6430 cpu_manager.go:414] "containerMap: removing container" podUID="07e2d7cb-e0d5-4b17-ba00-90bed359aa45" containerName="payload-pre-processing"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898369    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e2d7cb-e0d5-4b17-ba00-90bed359aa45" containerName="payload-pre-processing"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898420    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8b7c20-5a9f-4b7d-b1ad-c3405bd8be13" containerName="maas-api"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898427    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898433    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee9c9f4-6733-44c8-989a-abb5d9a1517e" containerName="istio-proxy"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898440    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a146bc62-fe6f-436a-bdb4-8128ee84731f" containerName="payload-processing"
Nov 03 08:46:26.898448 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.898446    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e2d7cb-e0d5-4b17-ba00-90bed359aa45" containerName="payload-pre-processing"
Nov 03 08:46:26.901493 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.901474    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:26.905441 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.905420    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsk8v\"/\"openshift-service-ca.crt\""
Nov 03 08:46:26.905564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.905452    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsk8v\"/\"kube-root-ca.crt\""
Nov 03 08:46:26.905564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.905474    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lsk8v\"/\"default-dockercfg-lfs26\""
Nov 03 08:46:26.912387 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.912367    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/must-gather-dtd55"]
Nov 03 08:46:26.934528 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.934499    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:46:26.986968 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.986929    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzwj\" (UniqueName: \"kubernetes.io/projected/38c52897-209d-4bd1-ac40-3da881101bdb-kube-api-access-bqzwj\") pod \"must-gather-dtd55\" (UID: \"38c52897-209d-4bd1-ac40-3da881101bdb\") " pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:26.987159 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:26.987036    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38c52897-209d-4bd1-ac40-3da881101bdb-must-gather-output\") pod \"must-gather-dtd55\" (UID: \"38c52897-209d-4bd1-ac40-3da881101bdb\") " pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:27.063875 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.063850    6430 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log" limit=1024
Nov 03 08:46:27.087954 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.087930    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzwj\" (UniqueName: \"kubernetes.io/projected/38c52897-209d-4bd1-ac40-3da881101bdb-kube-api-access-bqzwj\") pod \"must-gather-dtd55\" (UID: \"38c52897-209d-4bd1-ac40-3da881101bdb\") " pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:27.088107 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.087977    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38c52897-209d-4bd1-ac40-3da881101bdb-must-gather-output\") pod \"must-gather-dtd55\" (UID: \"38c52897-209d-4bd1-ac40-3da881101bdb\") " pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:27.088305 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.088286    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38c52897-209d-4bd1-ac40-3da881101bdb-must-gather-output\") pod \"must-gather-dtd55\" (UID: \"38c52897-209d-4bd1-ac40-3da881101bdb\") " pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:27.096444 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.096419    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzwj\" (UniqueName: \"kubernetes.io/projected/38c52897-209d-4bd1-ac40-3da881101bdb-kube-api-access-bqzwj\") pod \"must-gather-dtd55\" (UID: \"38c52897-209d-4bd1-ac40-3da881101bdb\") " pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:27.215762 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.215736    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/must-gather-dtd55"
Nov 03 08:46:27.486443 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.486350    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/must-gather-dtd55"]
Nov 03 08:46:27.490526 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:46:27.490493    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38c52897_209d_4bd1_ac40_3da881101bdb.slice/crio-6853eb954e5a301daf92528454bb74a0b1a1b5164089c2347e9acabd174a7b21 WatchSource:0}: Error finding container 6853eb954e5a301daf92528454bb74a0b1a1b5164089c2347e9acabd174a7b21: Status 404 returned error can't find the container with id 6853eb954e5a301daf92528454bb74a0b1a1b5164089c2347e9acabd174a7b21
Nov 03 08:46:27.492417 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.492395    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:46:27.594119 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:27.594086    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/must-gather-dtd55" event={"ID":"38c52897-209d-4bd1-ac40-3da881101bdb","Type":"ContainerStarted","Data":"6853eb954e5a301daf92528454bb74a0b1a1b5164089c2347e9acabd174a7b21"}
Nov 03 08:46:28.377306 ip-10-0-140-232 kubenswrapper[6430]: E1103 08:46:28.377272    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfvbs" containerName="extractor"
Nov 03 08:46:28.620281 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:28.620244    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/must-gather-dtd55" event={"ID":"38c52897-209d-4bd1-ac40-3da881101bdb","Type":"ContainerStarted","Data":"fc350e794c1946336e59acad94395fe498a504dd5c908d6b5517c11e9eb85109"}
Nov 03 08:46:28.620564 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:28.620541    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/must-gather-dtd55" event={"ID":"38c52897-209d-4bd1-ac40-3da881101bdb","Type":"ContainerStarted","Data":"dcac74f649a406b283c764f029b4bb9895fcc772673d27eec5a19c18ce525c55"}
Nov 03 08:46:28.634595 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:28.633117    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lsk8v/must-gather-dtd55" podStartSLOduration=1.857086867 podStartE2EDuration="2.633097954s" totalImagesPullingTime="776.011087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:28.632188217 +0000 UTC m=+3138.135023731" watchObservedRunningTime="2026-11-03 08:46:28.633097954 +0000 UTC m=+3138.135933466"
Nov 03 08:46:29.853375 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:29.853285    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-898qp_342d6281-4bdd-4090-ac8b-717ca2942a0a/global-pull-secret-syncer/0.log"
Nov 03 08:46:30.085033 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:30.085002    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-254xf_4ecea087-8175-4789-a497-7beeb69e1610/konnectivity-agent/0.log"
Nov 03 08:46:30.246117 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:30.246086    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-232.us-east-2.compute.internal_40f16205502b8d2684b7e492677380c9/haproxy/0.log"
Nov 03 08:46:37.417087 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:37.416556    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-q2k2f_145aee33-7914-4590-bc10-6c79746d247e/manager/0.log"
Nov 03 08:46:37.460726 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:37.460681    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qp7tx_3c8d6356-3592-4952-846d-3b7c992b6a3c/kuadrant-console-plugin/0.log"
Nov 03 08:46:38.773095 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:38.773041    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_679d7319-fe60-433b-b49f-42862dedd70b/alertmanager/0.log"
Nov 03 08:46:38.801055 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:38.801014    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_679d7319-fe60-433b-b49f-42862dedd70b/config-reloader/0.log"
Nov 03 08:46:38.823246 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:38.823220    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_679d7319-fe60-433b-b49f-42862dedd70b/kube-rbac-proxy-web/0.log"
Nov 03 08:46:38.847043 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:38.847020    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_679d7319-fe60-433b-b49f-42862dedd70b/kube-rbac-proxy/0.log"
Nov 03 08:46:38.871301 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:38.871271    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_679d7319-fe60-433b-b49f-42862dedd70b/kube-rbac-proxy-metric/0.log"
Nov 03 08:46:38.910084 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:38.908431    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_679d7319-fe60-433b-b49f-42862dedd70b/prom-label-proxy/0.log"
Nov 03 08:46:38.932602 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:38.932574    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_679d7319-fe60-433b-b49f-42862dedd70b/init-config-reloader/0.log"
Nov 03 08:46:39.100604 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.100523    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-688749c497-5dxnm_9600bd0a-077f-40c5-b09d-eac5da61049b/metrics-server/0.log"
Nov 03 08:46:39.235825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.235782    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vfxjv_8387d126-b80b-43e1-978b-4ee863c4ddee/node-exporter/0.log"
Nov 03 08:46:39.264088 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.260759    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vfxjv_8387d126-b80b-43e1-978b-4ee863c4ddee/kube-rbac-proxy/0.log"
Nov 03 08:46:39.287331 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.287308    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vfxjv_8387d126-b80b-43e1-978b-4ee863c4ddee/init-textfile/0.log"
Nov 03 08:46:39.388529 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.388459    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-98nq2_c8b98521-a087-4c0d-a301-a49080ff65b5/kube-rbac-proxy-main/0.log"
Nov 03 08:46:39.413676 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.413648    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-98nq2_c8b98521-a087-4c0d-a301-a49080ff65b5/kube-rbac-proxy-self/0.log"
Nov 03 08:46:39.435367 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.435338    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"]
Nov 03 08:46:39.441176 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.441153    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.443622 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.443593    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-98nq2_c8b98521-a087-4c0d-a301-a49080ff65b5/openshift-state-metrics/0.log"
Nov 03 08:46:39.450825 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.450639    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"]
Nov 03 08:46:39.552658 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.552617    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-lib-modules\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.552865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.552672    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-proc\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.552865 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.552842    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-podres\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.552986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.552876    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-sys\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.552986 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.552918    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7t6\" (UniqueName: \"kubernetes.io/projected/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-kube-api-access-hv7t6\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654550 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654465    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-proc\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654553    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-podres\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654588    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-sys\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654607    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-proc\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654642    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7t6\" (UniqueName: \"kubernetes.io/projected/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-kube-api-access-hv7t6\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654690    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-podres\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654693    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-sys\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.654743 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654704    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-lib-modules\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.655101 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.654782    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-lib-modules\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.683699 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.683667    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7t6\" (UniqueName: \"kubernetes.io/projected/4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e-kube-api-access-hv7t6\") pod \"perf-node-gather-daemonset-qbscm\" (UID: \"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.759395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.759355    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:39.797195 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.797144    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66b6b9b7d5-vw49l_37e9ee43-9770-42c8-9064-92437099e68a/telemeter-client/0.log"
Nov 03 08:46:39.825229 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.820900    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66b6b9b7d5-vw49l_37e9ee43-9770-42c8-9064-92437099e68a/reload/0.log"
Nov 03 08:46:39.848590 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:39.848431    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66b6b9b7d5-vw49l_37e9ee43-9770-42c8-9064-92437099e68a/kube-rbac-proxy/0.log"
Nov 03 08:46:40.045898 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:40.045864    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"]
Nov 03 08:46:40.053034 ip-10-0-140-232 kubenswrapper[6430]: W1103 08:46:40.053008    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4f2e2e6a_2cdf_4931_8fdb_6caabb79db2e.slice/crio-4361163a89d8894e82ce21467919a1fb3523e9d3580fd1abf770a832b582f17b WatchSource:0}: Error finding container 4361163a89d8894e82ce21467919a1fb3523e9d3580fd1abf770a832b582f17b: Status 404 returned error can't find the container with id 4361163a89d8894e82ce21467919a1fb3523e9d3580fd1abf770a832b582f17b
Nov 03 08:46:40.752082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:40.746979    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm" event={"ID":"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e","Type":"ContainerStarted","Data":"e6697857623b079d280d4a5333ea867e752b5cfd10812cf2100d99281f2c7369"}
Nov 03 08:46:40.752082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:40.747046    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm" event={"ID":"4f2e2e6a-2cdf-4931-8fdb-6caabb79db2e","Type":"ContainerStarted","Data":"4361163a89d8894e82ce21467919a1fb3523e9d3580fd1abf770a832b582f17b"}
Nov 03 08:46:40.752082 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:40.747890    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:40.766306 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:40.764285    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm" podStartSLOduration=1.764266911 podStartE2EDuration="1.764266911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:40.763245375 +0000 UTC m=+3150.266080890" watchObservedRunningTime="2026-11-03 08:46:40.764266911 +0000 UTC m=+3150.267102426"
Nov 03 08:46:43.814871 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:43.814837    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4p5vg_bb54163a-2ad1-4b02-93d6-f8eac877da15/dns/0.log"
Nov 03 08:46:43.837185 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:43.837158    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4p5vg_bb54163a-2ad1-4b02-93d6-f8eac877da15/kube-rbac-proxy/0.log"
Nov 03 08:46:43.940940 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:43.940909    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7zvx_16c1f0f2-b9b6-4659-97fb-9b3092e04f5f/dns-node-resolver/0.log"
Nov 03 08:46:44.612574 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:44.612533    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-srvsw_e18c61c8-db8c-49ca-8fbd-9378be7e9c51/node-ca/0.log"
Nov 03 08:46:46.589952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:46.589918    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-2hhrl_e23318d3-5990-4aea-9c49-e821bb20fd56/kube-auth-proxy/0.log"
Nov 03 08:46:46.649189 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:46.649161    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-66kgk_7a5a8bac-1e6a-4fb8-9852-abc001854117/istio-proxy/0.log"
Nov 03 08:46:46.738179 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:46.738139    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bchrg_f2b07e41-a95f-452b-b166-c4532a2daee0/istio-proxy/0.log"
Nov 03 08:46:46.857696 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:46.857613    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-s92vd_6ef2358a-4551-4237-bed0-4ad8b79652bd/istio-proxy/0.log"
Nov 03 08:46:47.770520 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:47.770492    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-qbscm"
Nov 03 08:46:49.076395 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:49.076365    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-45pld_d680dcf8-8cd0-488b-b8b1-b66368817faf/serve-healthcheck-canary/0.log"
Nov 03 08:46:49.753022 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:49.752982    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfvbs_de01be03-1a51-4ada-9b3c-498276ad5677/kube-rbac-proxy/0.log"
Nov 03 08:46:49.778995 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:49.778050    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfvbs_de01be03-1a51-4ada-9b3c-498276ad5677/exporter/0.log"
Nov 03 08:46:49.806500 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:49.806425    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfvbs_de01be03-1a51-4ada-9b3c-498276ad5677/extractor/0.log"
Nov 03 08:46:50.684496 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:50.684462    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-2hhrl_e23318d3-5990-4aea-9c49-e821bb20fd56/kube-auth-proxy/0.log"
Nov 03 08:46:50.720414 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:50.720383    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-66kgk_7a5a8bac-1e6a-4fb8-9852-abc001854117/istio-proxy/0.log"
Nov 03 08:46:50.796137 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:50.796109    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bchrg_f2b07e41-a95f-452b-b166-c4532a2daee0/istio-proxy/0.log"
Nov 03 08:46:50.929252 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:50.929224    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-s92vd_6ef2358a-4551-4237-bed0-4ad8b79652bd/istio-proxy/0.log"
Nov 03 08:46:52.779103 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:52.779079    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6dd9d888fc-xwwvj_19c36ccd-5593-4a5d-9c5b-4d60c58dab44/manager/0.log"
Nov 03 08:46:52.803900 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:52.803867    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6dd9d888fc-xwwvj_19c36ccd-5593-4a5d-9c5b-4d60c58dab44/copy-manifests/0.log"
Nov 03 08:46:54.566848 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:54.566816    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-v6gdq_35013bdc-2709-4435-8e0f-cbb535870313/manager/0.log"
Nov 03 08:46:54.645925 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:54.645889    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p6rtl_5dc72eb0-6403-491f-bb27-dd3ccc6a2cc1/manager/0.log"
Nov 03 08:46:55.093255 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:55.093223    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-2hhrl_e23318d3-5990-4aea-9c49-e821bb20fd56/kube-auth-proxy/0.log"
Nov 03 08:46:55.134167 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:55.134134    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-66kgk_7a5a8bac-1e6a-4fb8-9852-abc001854117/istio-proxy/0.log"
Nov 03 08:46:55.198878 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:55.198817    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bchrg_f2b07e41-a95f-452b-b166-c4532a2daee0/istio-proxy/0.log"
Nov 03 08:46:55.293778 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:55.293727    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-s92vd_6ef2358a-4551-4237-bed0-4ad8b79652bd/istio-proxy/0.log"
Nov 03 08:46:55.372439 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:55.372354    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-r57t2_862e4682-de0b-4e34-9bb8-626f0ab19a83/manager/0.log"
Nov 03 08:46:55.458952 ip-10-0-140-232 kubenswrapper[6430]: I1103 08:46:55.458897    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799454fcf4-2rncm_25c36014-1407-4536-ad79-b3b4e16be4c0/manager/0.log"